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EXECUTIVE  SUMMARY 


OBJECTIVES 

The  work  presented  in  this  report  was  performed  under  Subtask  2  and  3  of  Task  Order  33  of 
the  SDIO  Systems  SETA  contract.  The  purpose  of  these  subtasks  was  to  evaluate  current  software 
measurement  processes  and  to  evaluate  current  software  measurement  support  tools.  Teledyne 
Brown  Engineering  was  the  technical  lead  on  the  subtasks  with  support  from  Sparta  and  TASC. 

The  subtasks  consisted  of  the  following  activities: 

o  collection  of  extensive  metrics  data  from  industry.  Government  and  academic 
sources; 

o  evaluation  and  analysis  of  the  collected  information  for  specific  relevancy  within  the 
SDIO  software  domain  iqjplications; 

o  review  of  tools  databases  and  product  information  literature  to  identify  potential 
metrics  tools  applicable  to  SDI  software  support; 

o  identification  of  deficiencies  or  domain  limitation  of  existing  methods  and  models. 


CONCLUSIONS 

There  are  several  major  conclusions  from  this  subtask.  The  results  of  our  review  of  available 
and  ongoing  metrics  programs  reveal  that  a  metrics  methodology  is  needed  for  the  effective  use  of 
metrics  and  each  major  system,  development/acquisition  must  develop  and  tailor  its  own  metrics 
program.  SDI  is  no  exception.  There  is  a  need  to  emphasize  predictive  measurands  in  the  earlier 
phases  of  the  life  cycle  (i.e.,  requirements  and  design).  In  the  near  term,  available  metrics  tool 
packages  can  be  halpful,  but  greater  use  of  formal  notation  to  support  requirements  and  design 
synthesis  is  required  if  metrics  information  rigor  and  application  is  to  occur  in  a  much  more 
disciplined  and  predictive  manner. 


OPEN  ISSUES 

The  selection  of  application  data  used  with  metrics  models  must  be  carefully  selected,  scaled 
and  scoped.  Formal  metrics  models  for  use  in  the  early  life  cycle  phases  must  be  developed.  In 
turn,  a  life  cycle  model,  iterative  in  nature,  must  be  identified  that  can  successfully  be  used  for  SDI 
development.  Without  a  proper  life  cycle  and  metric  requirements  model,  the  state-of-the-art  in 
predictive  metrics  will  continue  to  lag  behind  post-facto  metrics  for  some  time  to  come.  Predictive 
metrics  are  essential  if  higher  produaivity  yields  and  better  quality  reusable  software  is  to  become  a 
reality. 
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00. 


INTRODUCTION 


This  report  provides  an  evaluation  of  current  software  measurement  processes  and  tools 
currently  used  in  the  Government,  industry  and  academia. 

Section  1  lists,  summarizes  and  analyzes  the  metrics  models  from  available  databases  and 
software  documents.  Section  2  presents  an  assessment  of  how  each  applicable  software  metric  can 
be  applied  within  each  SDS  software  subfunction.  For  each  candidate  software  metric  identified,  a 
validation  analysis  summary  will  be  presented.  Section  3  presents  the  results  of  analyzing  the  field 
experience  with  two  initiatives  highlighted.  Section  4  identifies  capabilities  and  limitations  in 
current  methods.  Section  5  presents  a  literature  and  database  survey  identifying  existing  metrics 
tools  and  environments. 

The  collection  of  metrics  data  proved  to  be  both  skewed  and  elusive.  Skewed  in  the  sense 
that  much  of  the  existing  metrics  information  that  is  found  to  exist  and  is  formalized  via  models  and 
mathematical  relationships  is  encountered  late  in  the  life  cycle  (i.e.,  occurring  in  the  implementation 
phase  and  beyond).  Elusive  due  to  the  fact  that  early  life  cycle  (predictive)  metrics  are  virtually 
non-existent,  and  when  identified,  have  no  formal  foundations  (mathematical  or  relational)  to 
suppon  them  for  the  most  part.  Furthermore,  consensus  is  still  involing  on  the  relative  value  of 
specific  metrics  (e.g.,  complexity)  amoung  metrics  "experts”.  Essentially  each  program  must 
establish  and  implement  its  own  metrics  and  quality  program  to  derive  maximum  benefit  from  it. 

A  section  on  life  cycle  models  (1.6)  has  been  incorporated  into  this  report.  This  section  is 
intended  to  support  the  need  for  a  new  life  cycle  model  consistent  with  the  MIL-STD-2167A 
requirements.  It  is  also  intended  to  provide  further  insight  on  the  need  for  metrics  emphasis  and 
their  relationships  in  the  initial  (early)  life  cycle  phases,  and  the  development  of  more  formal 
predictive  metrics.  Such  emphasis  and  identification  appears  to  provide  a  key  component  to 
achieving  higher  productivity  and  quality  thresholds. 

Some  of  the  data  collected  and  conclusions  arrived  at,  while  disappointing  due  to  the  state-of- 
the-metric-art,  are  not  surprising  and  confirm  expectations.  However,  the  life  cycle  section, 
together  with  the  evidence  identified  in  section  2.1,  giving  support  to  a  multi-attribute,  composite 
or  vector  metric  representation  presents  a  potential  breakthrough  in  the  SOA.  The  latter  metrics 
form  is  the  subject  of  the  1990  International  Conference  on  Metrics  being  held  in  the  United  States 
for  the  first  time  in  several  years. 

With  respect  to  a  tools/environment  database,  the  Army’s  TASQ  database  contains  a  wealth 
of  information  as  evidenced  by  Appendix  C  and  D,  and  has  served  as  an  extremely  valuable  source 
of  information,  representing  the  latest  and  most  extensive  survey  in  the  marketplace. 

This  document  is  driven  by  the  software  measurement  requirements  established  in  the  Task 
Order  33,  TR-9033-1,  and  is  consistent  in  the  use  of  the  software  domain  and  function 
identifications  of  that  document.  The  document  also  extends  and  complements  that  information  of 
TR-9033-1. 


1 


TAScf 

TVC 
•PAKTA 
AMI  1 
SMC 
BSA  ! 

^  ^ 

1 

» 

TATUM 

mo 

THE  ANALYTIC  SCIENCES  CORPORATION 


1 .  REVIEW  OF  EXISTING  METRICS 


The  overriding  theme  for  this  survey  and  evaluation  of  applicable  software  metrics  is  to 
point  up  practical  tools  for  acquisition  and  developmental  program  managers  to  make  appropriate 
estimations  and  useful  assessments  on  real-world  problems.  As  Dr.  J.  Short  of  the  Naval 
Underwater  Systems  Command  indicated,  "we  get  results  from  our  metrics  or  we  change  them." 

I.l  LITERATURE  SURVEY 

Priortizing  the  metrics  literature  reading  of  a  project  manager  or  V&V  planner  is  a  first  step 
in  managing  a  practical  metrics  program.  The  key  concepts  of  quality  and  productivity  factors  and 
criteria  within  the  project  life  cycle  and  framework  provide  the  ftmdamentals  for  solid  plaiming. 

I.l.  I  Framework  Discussions 

An  understanding  of  the  structure  of  the  software  metrics  is  facilitated  by  the  definition  of 
the  software  quality  metrics  framework.  Though  introduced  by  [RADC8502],  the  IEEE  draft 
Standard  for  Software  Quality  Metrics  has  refined  the  framework. 

The  quality  metrics  framework  provides  an  open-ended,  hierarchy  for  organizing  the 
conceptual  elements  of  the  quality  metrics  domain.  These  elements  are  quality  attributes,  criteria 
and  measurements  suitable  for  organizing  quality  control  rooms  and  management  tracking.  The 
IEEE  refinements  include  the  concepts  of  "direct  metrics"  (for  quality  factors  like  reliability)  and 
"subfactors"  (e  g.,  for  corrccmess).  (Refer  to  Figure  1-1) 

[IEEE  (draft)  P 1061] 

Standard  for  a  Software  Quality  Metrics  Methodology 

Section  3  presents  management-oriented  objectives. 

Section  4  presents  the  refuted  Software  (Quality  Metrics 
Framework. 

A  good,  comprehensive  graduate  text  on  software  quality  metrics  was  written  by  S.D. 
Conte,  H.E.  Dunsmore  and  V.Y.  Shen  in  a  collaborative  effort  between  Purdue  University  and  the 
Microelectronics  and  Computer  Technology  Corporation  (MCC).  The  authors  present  many 
measurement  and  analysis  examples. 

[CONT86J 

Software  Engineering  Metrics  and  Models.  Benjamin  Cummings,  1986. 

Also,  more  panicular  to  SDIO  and  the  SDS,  the  (draft)  IDA  paper  P-2132  (draft)  [IDA8812] 
present  a  terser  introduction  that  would  be  suitable  for  an  SDS  management  pamphlet. 

[IDA  P-2132  J 

SDS  Software  Testing  and  Evaluation:  A  Review  of  the  State-of-the-Ait  in  Software 
Testing  and  Evaluation.  Chapter  6.  "Software  Measurement  Technology"  present  a  current  review 
of  the  state  of  the  art  of  software  quality  metrics  mediodology. 
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REFINED  SOFTWARE  QUALITY  METRICS  FRAMEWORK 


SOFTWARE  QUALITY 
OF  SYSTEM  X 


FACTOR 

FACTOR 

FACTOR 

Direct  Metric 

Direct  Metric 

Direct  Metric 

SUB-FACTOR 

SUB-FACTOR 

SUB-FACTOR 

METRIC 


METRIC 


IEEE  P1061  (DRAFT) 


FIGURE  1-1 


TVK 
•FAMTA 
AHI  I 

:  one  ! 

PAA  ! 


IHI  •mOMYID  «r  A  YUM 
eONMmDTOSM 


1-2 


THE  ANALYTIC  SCIENCES  CORPORATION 


Chapter  7,  "Software  Reliability  Assessment  Technology"  presents  a  concise,  yet  relevant 
discussion  of  the  potential  shortfall  in  current  reliability  metrics  for  the  SDS  software  metrics 
requirements. 

[RADC-TR-85-37.  3  vols] 

Specification  of  Software  Quality  Attributes.  Vol.  1  introduces  the  framework  and 
concepts  of  qual'ty  specification  and  evaluation.  Volume  2  is  a  detailed  guidebook  for 
specification.  Vomme  3  is  the  evaluation  guide  with  sample  checklists  and  worksheets. 

[RADC-TR-87-171  \l] 

Methodology  for  Software  Reliability  Prediction  and  Assessment.  3.1  "Software  Quality 
Measurement  Framework"  includes  the  original  framework  discussion  with  definitions  of  all 
reliability  concepts. 

1.1.2  Management  and  Quality  Indicators  and  Factors 

The  Air  Force  pamphlets  AFSCP-800-14  and  AFSCP-800-43  present  a  solid  introduction 
to  management  (performance)  and  quality  indicators.  The  pamphlets  correlate  the  management  and 
quality  indicators,  and  then  each  quality  indicator  to  its  applicable  software  quality  factors  (as 
introduced  by  [RADC8502].  One  disturbing  statement  "...there  are  no  widespread  tools 
available...”  depends  on  the  writers  concept  of  "widespread",  and  is  misleading  -  tools  are 
available. 

The  quality  indicators  pamphlets  [-800-14]  buries  an  important  concept  for  management 
among  two  graphs  and  a  chart: 

4-13....  the  degree  of  management  insight  can  be  significantly  increased  by  using  the 
software  quality  indicators.  This  combination  of  management  and  quality  indicators  should  enable 
the  contractor  and  the  SPO  to  manage  software  development  activities  actively  instead  of  reacting  to 
software  crises  as  they  arise... 

Note:  The  Aimy  has  replicated  the  Air  Force  pamphlets  with  few  variations. 

[AFSCP-800-14]  Software  Quality  Indicators 
[AMCP-70-I4J 

[AFSCP-800-43]  Software  Management  Indicators 
[AMCP-70-13] 

Figure  1-2  presents  the  correlations  of  factors  and  indicators  as  per  the  pamphlets. 

1.1.3  Management  Methodology 

As  their  draft  evolves,  Norman  Schneidewind  (USNPGS)  and  others  on  the  IEEE  Quality 
Metrics  Standard  Committee  have  emphasized  the  development  of  an  extensible  metrics 
management  method  for  each  software  acquisition  or  evaluating  organization.  The  approach  taken 
by  Dr.  John  Short  and  his  metrics  staff  at  NUSC  is  similar,  emphasizing  clear,  well-understood 
metrics  objectives,  planning  at  the  single  project  level  with  SOWs  tailored  to  meet  life  cycle 
assessments  points.  The  Air  Force  methodology  guide  emphasizes  iterative  planning,  predicting 
and  assessment.  [IEEE(d)Pl()61],  [RADC878aM] 
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Indicators 


Software  Management  and  Quality  Indicators  [APSCP  800*14] 

Management  Indicators 
CRU  SDM  RD4SSPDT  C/SD  SDT 


Completeness 
Design  Structure 
Defect  Density 
Fault  Density 
Test  Coverage 
Test  Sufficiency 
Documentation 


CRU  *  Computer  resource  utilization 
SDM  «  Software  development  manpower 
RD&S  >  Requirements  definition  &  stability 
SPDT«  Software  progress  ■  development  &  test 
C/SD  «  Cost/Schedule  Deviations 
SDT  »  Software  development  tools 


Quality 

Indicators 

Completeness 
Design  Structure 
Defect  Density 
Fault  Density 
Test  Coverage 
Test  Sufficiency 
Documentation 


Software  Quality  Indicators  and  Factors  [AFSCP  800*14] 

Software  Quality  Factors 

Corr  Effc  Flex  Intg  Into  Main  Port  Reli  Reus  Test  Usab 


Corr  -Correctness 
Effc  -  Efficiency 
Flex  -  Flexibility 
Intg  -  Integrity 
Into  -Interoperability 


Main  -Maintainability 
Port  -  Portability 
Reli  -  Reliability 
Reus  -Reusability 
Test  -Testability 
Usab  -Usability 


Figure  1*2.  Software  Management  /Quality  Indicators  &  Factors 
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(Further  discussion  of  Life  Cycle  importance  is  in  Paragraph  1 .6.  The  NUSC  environment 
is  discussed  in  Paragraph  3.1.) 

i.1.4  Reliability 

As  Musa,  lannino  and  Okumoto  introduce,  "Reliability  is  probably  the  most  important  of 
the  characteristics  "...of  software  quality.  They  define  reliability  traditionily  "the  probability  that 
the  software  will  work  without  failure  for  a  period  of  time  to  meet  customer  requirements."  This 
subsumes  many  properties  of  software  quality  (correcmess,  friendliness,  safety,...)  but  excludes 
modifiability,  readibUity  (maintainability),  which  are  less  quantifiable. 

[MUSA87] 

Software  Reliability  -  Measurement.  Prediction.  Application.  McGraw  Hill,  1987. 

This  comprehensive  text  provides  practical  measurement  ^plication  guidance  problems  and 
solutions  for  managers  and  practicing  software  engineers,  as  well  as  the  theoretical  discussions 
needed  for  a  college  course.  It  uses  the  traditional  hardware  reliability  approach  taking  advantage 
of  the  body  of  systems  and  control  reliability  knowledge.  There  is  a  reliance  on  probabilistic 
projections  using  random/stochastic  techniques. 

A  key  point  about  this  approach;  It  is  based  upon  unpredictability  of  programmer  errors 
and  unpredictability  of  execution  conditions  complicated  by  machine  states.  This  does  not  account 
for  the  "known  error"  category  (non-fatal,  moderately  reducing  faults).  Random  implies 
"unpredictable”  vs.  "uniform".  The  use  of  techniques  based  upon  this  randomness  assumption 
should  not  preclude  other  techniques  which  would  presume  some  predictability  about  the  failures, 
faults  or  defects. 

Some  further  points  made  by  John  Musa  in  his  articles  should  also  be  noted: 

a.  Reliability  based  on  failure  statistics  is  user  oriented,  more  significant  and  suitable 
for  setting  (user-oriented)  objectives  (for  prudent  business  services  and  typical 
information  systems). 

b.  Defect/correction  based  quality  measures  are  only  significant  in  terms  of  contractor 
performance  toward  target  goals. 

Other  key  works  on  software  reliability  are  from  the  Air  Force  RADC  and  the  IEEE.  The 
RADC  work  provides  detail  profiling  of  reliability  and  testing  measurements.  The  more  recent 
work  of  the  IEEE  Software  Reliability  Measurement  Working  Group  is  providing  a  refined 
dictionary  and  guidebook  for  reliability,  including  detail  directory  lookup  for  individual  software 
metric  parameters,  a  cohesive  notation,  instructions  and  evaluation  references. 
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[RADC-TR-87-171] 

Methodology  (Vol.  1)  and  Guidebook  (Vol.  2)  for  Software  Reliability  Prediction  and 
Estimation 

[IEEE  982. 1 J 

(Draft)  Standard  Dictionary  of  Measures  to  Produce  Reliable  Software 
[IEEE  982.2] 

(Draft)  Guide  for  the  Use  of  the  Standard  Dictionary  of  Measures  to  Produce  Reliable 
Software 


As  preyiously  mentioned,  the  IEEE  Quality  Metrics  Standard  Committee  is  proyiding 
standard  methodology  with  appendices  of  measurements,  experience  reports  and  validation  guides 
[IEEEP1061]. 

1.1.5  Aggregate  Indicators  vs.  Simple  Measurements 

The  Air  Force  pamphlets,  [CARD8707]  and  {GOIC89RC]  indicate  that  primitive  measures 
are  to  be  aggregated  with  weighting  factors  based  upon  the  relative  value  of  the  functions  or 
modules  effected  when  building  the  quality  indicators.  In  the  Waterfall  SDLC  these  weights  can  be 
derived  by  requirements  prioritization  at  the  top  level,  and  then  functional  factoring  as  the 
requirements  are  delineated.  However,  in  a  prototyping  development,  functional  and  performance 
gods  may  be  used  to  establish  relative  system  functional  values. 

Apart  from  the  goal-oriented  weights,  the  characteristics  of  the  software  architecture  affect 
the  relationship  between  software  elements.  John  Musa  has  recently  alerted  us  that  the  criticality  of 
functions  within  the  evolving  system  software  architecture  is  a  paramount  consideration  when 
considering  software  reliability,  especially  in  regard  to  high-use  servo  functions  or  boundary 
control  elements  (e.g..  Three  Mile  Island  fault). 

1.1.6  Validation  and  Refinement  Articles 

The  technical  literature  is  primarily  concerned  with  the  detail  validations  of  particular 
reliability  and  maintainability  assessment  measures  and  more  recently  has  been  focusing  upon  the 
validity  of  predictions.  There  seems  to  be  agreement  among  metrics  experts  that  users  should 
carefully  plan  and  build  understanding  of  the  unique  characteristics  of  their  software  and 
environment.  Most  of  the  software  measures  including  balances  or  weights  for  development 
experience  and  people  factors  have  shown  to  be  more  significant  than  unadjusted  measures  of  the 
software  alone. 


We  observed  that  the  complexity  and  productivity  models  and  tools  do  exist,  but  their 
interpretations  and  validations  vary.  Not  much  validation  has  been  done  on  maintainability 
although  tools  do  exist.  Other  issues  such  as  software  integrity,  portability,  usability,  and 
reusabUity  lack  tools  as  well  as  validations.  There  are  numerous  validations  and  reliability  models. 

Some  of  the  more  significant  results  reached  are  listed  below: 


[TAKA8901]  The  results  of  this  article  indicate  that  models  based  on  factors  such  as  frequency  of 
program  specification  change,  programmer's  skills,  and  volume  of  program  design  document  are 
more  reliable  than  conventional  error  prediction  methods  based  only  on  program  size. 
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[CARD8812]  The  results  of  this  article  indicate  that  complexity  indicators  based  on  structural 
(intermodule)  complexity  and  local  (intramodule)  complexity  seemed  to  be  a  useful  tool  for 
evaluating  design  quality  before  committing  the  design  to  code. 

[LEW8811]  The  results  of  this  article  indicate  that  complexity  measures  are  useful  in  quantifying 
the  design  and  provide  a  guide  to  designing  reliable  software. 

[BOEH8810J  Barry  Boehm  and  Philip  Papaccio  present  some  cost  and  quality  observations 
specificaUy  applicable  to  planned  rapid  deployment  prototyping  and  other  hybrid  life  cycle  notions; 
low  quality  -  low  reliability  software  costs  much  more  to  maintain;  high  quality  software  tends  to 
reduce  its  costs.  High  quality  and  low  costs  are  promoted  by  personnel  incentives,  work 
environment  and  tool  enhancements,  and  by  software  reuse  with  low  rework.  The  authors  cite  use 
of  their  database  of  projects  to  validate  the  COCOMO  model(s). 

[WEIS8810]  Weiss  and  Weyuker  established  an  extended  domain  model  of  software  reliability  and 
used  the  lannino,  Musa.  Okumuto  rating  criteria  [IANN84].  The  significance  is  that  the  notion  of 
tolerable  discrepancies  of  user  service  are  introduced  into  a  refined  definition  of  software 
reliability,  and  the  domain  of  program  performance  is  enforced.  This  work  was  analytical  and 
preliminary  and  needs  followup,  but  promises  applicability  to  large  systems  like  the  SDS,  where 
the  criticality  of  failures,  the  severity  of  defects,  and  the  tolerance  levels  should  be  factored  into 
reliability  measures.  The  second  author,  Elaine  J.  Weyuker  has  independent  software  reliability 
work.  Both  were  at  NYU. 

[DAVI8809]  This  short  article  analyzed  the  applicability  of  complexity  measures  (McCabe, 
Halstead)  and  lines  of  code  in  contrast  to  a  group  of  new  measures  which  analyze  computer 
programs  in  terms  of  cognitive  "chunks"  as  proposed  by  Lamergan,  Mayer,  Schneiderman  and 
Solway  (in  various  papers)  "the  software  psychologists".  The  validating  scope  was  limited  to  a  set 
of  small  Fortran  programs.  Some  significant  observations  are: 

a.  Halstead's  length  oriented  "E"  measure  is  more  robust  than  anticipated. 

b .  Good  debug  time  predictors  also  predicted  the  error  counts  well. 

c .  Constructive  time  for  new  programs  depends  more  on  size  than  complexity. 

d .  Maintenance  time  is  strongly  related  to  program  complexity  and  lesser  to  overall 
size. 

e .  Data  structure  complexity  is  more  significant  to  maintenance  effort. 

f .  Control  flow  complexity  is  more  significant  to  constmctive  effort. 

g.  Reinforces  the  view  of  Conte,  Dunsmore  and  Shen  that  early  SDLC  phase  metadata 
is  fairly  accurate  at  defining  the  actual  data  complexity  developed.  [CONTE8401] 

[RAMA8808]  This  was  a  successful  combination  of  Halstead's  Software  Science  measures  with 
McCabe's  Cyclomatic  Complexity  Measure,  using  a  set  of  weighting  factors  for  the  length  and 
volume  primitives.  However,  the  combined  results  have  no  greater  validity  than  the  simple 
combination  of  separate  measures. 
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[CARDS?  12]  This  article  demonstrated  that  the  basic  relation  of  software  science  lacks  empirical, 
as  well  as  theoretic  support.  Future  models  of  program  constmction  process  should  be  based  more 
closely  on  observations  of  what  programmers  actually  do. 

[JEFF8707]  This  article  found  that  the  complex  relationship  between  effort  (productivity)  and 
elapsed  time  variation  cannot  be  measured  by  any  of  the  current  time-sensitive  cost  models. 

[PERK8703]  This  article  investigated  a  Navy-supplied  Ada  software  and  showed  that  an 
automatable,  hierarchical.  Ada-specific  software  metrics  framework  is  an  effective  aid  for 
improving  the  quality  of  Ada  software. 

[ABDE8609]  This  article  examined  10  metrics;  Jelinski  and  Moranda,  Bayesian  Jelinski- 
Moranda,  Littlewood,  Bayesian  Littlewood,  Littlewood  and  Verrall,  Keiller  and  Littlewood, 
Weibull  Order  Statistics,  Duane,  Goel-Okumoto  Model,  and  Littlewood  NHPP  model.  The  goal 
of  the  paper  is  to  present  an  approach  in  deciding  which  is  the  most  appropriate  model  to  use.  The 
results  showed  that  there  is  no  "best  buy"  among  the  10  metrics.  This  is  because  predictive 
measures  perform  with  varying  degrees  of  accuracy  on  different  software  being  studied. 
Therefore,  the  users  need  to  be  able  to  select  and  be  sure  that  a  chosen  prediction  metric  is 
perfonning  well  for  the  type  of  prediction  required. 

[ALBR8411]  This  article  demonstrates  the  equivalence  between  Albrecht's  methodology  to 
estimate  the  amount  of  the  "function"  or  "function  points"  the  software  is  to  perform,  and 
Halstead's  "software  science"  model  of  "SLOC"  measure.  It  also  demonstrates  the  high  degree  of 
correlation  between  "function  points"  and  the  "SLOC"  (source  lines  of  code)  of  the  program,  and 
between  "function  pwints"  and  the  work-effort  required  to  develop  the  code. 

[BASI831  lA]  This  paper  attempts  to  validate  the  Halstead's  Software  Sciences  metrics,  McCabe's 
Cyclomatic  Complexity  and  other  standard  programs  measures.  The  results  of  this  article  indicate 
that  the  Software  Science  metrics,  Cyclomatic  Complexity  and  other  predictive  metrics  do  not 
satisfactorily  measure  the  errors  occurred  during  development,  and  neither  of  these  metrics  really 
measure  or  predict  effort  or  quality. 

[HENR81 J  The  article  showed  that  the  measurement  of  software  quality  for  large-scale  systems 
using  information  flow  to  represent  the  system  interconnectivity  is  an  important  and  viable 
technique. 

1.1.7  Summari/ing  the  Validation  Literature 

[LEW8811]  and  [CARD88I2]  agreed  that  complexity  measures  are  useful  in  measuring 
and  quantifying  the  design  effort  and  provide  a  guide  to  designing  reliable  and  high-quality 
software.  In  contrast,  [BASI831  lAJ  showed  that  the  software  science,  cyclomatic  complexity 
metrics  and  other  predictive  metrics  do  not  satisfactorily  measure  the  errors  incurred  during  the 
development  phase,  and  neither  of  these  metrics  really  measures  or  predicts  design  effort  or 
software  quality.  Ramamurthy  and  Melton  [RAMA8808]  examined  software  science  and 
cyclomatic  complexity  and  observed  that  combining  these  measures  was  as  effective  as  making  the 
separate  assessments.  They  proposed  a  family  of  weighted  measures  which  simultaneously  detect 
the  software  characteristics  that  are  detected  by  the  software  science  measures  and  the  cyclomatic 
number.  [GIBS8903]  studied  the  effect  of  system  structure/complexity  on  system  maintainability; 
specifically,  what  effect  do  structural  differences  have  on  maintenance  performance,  and  are 
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Structural  differences  measurable?  The  results  indicate  that  the  structural  differences  do  impact 
performance  and  the  metrics  being  validated  (i.e.,  Halstead's  E,  McCabe's  v(G),  Woodward's  K, 
Gaffney's  Jumps,  Chen's  MIN  and  Benyon-Tinker's  C2)  can  be  used  as  project  management 
tools. 


From  the  article's  results,  some  conclusions  can  be  made: 

a.  There  is  an  evolving  consensus  among  complexity  metrics  validators  that  the 
application  data  used  with  the  complexity  models  must  be  carefully  selected  and  the 
conclusions  must  be  scoped  and  scaled. 

b.  The  validity  metrics  models  for  estimation  and  prediction  requires  continual 
updating. 

c .  There  is  a  need  to  emphasized  predictive  measurands  in  the  earlier  phases  of  the  life 
cycle  (i.e,,  requirements  and  design). 

d.  The  validation  process  must  continue  so  that  metrics  can  be  effectively  used  to 
characterize  and  evaluate  software  products  and  processes. 

e .  A  metrics  methodology  is  needed  for  the  effective  use  of  metrics  and  each  major 
system,  development/acquisition  must  develop  and  tailor  its  own  metrics  program. 


1.2  DATABASES 

Technical  interchanges  and  queries  have  been  performed  with  the  following  information 
sources: 

a.  The  TASQ  Repository  maintained  for  the  U.S.  Army  AMCCOM  Product 
Assurance  Directorate  in  New  Jersey.  This  database  contains  information  on 
several  thousand  tools,  characteristics  and  companies.  The  information  is  available 
via  a  series  of  in  depth  classification  matrices  and  vendor  descriptors.  Appendix  C 
contains  selected  portions  of  the  TASQ  database  with  potential  direct  or  indirect 
support  potential  for  the  SDS.  A  complete  TASQ  expansion  is  separately  bound  in 
single  copy  as  Appendix  D  (too  physically  large  for  reproduction). 

b.  The  Air  Force  Systems  Command  Rome  Air  Development  Center  at  GrifFiss  Air 
Force  Base,  New  York.  A  number  of  technical  reports  from  this  source  are 
referenced  in  the  document  list.  The  Data  Analysis  Center  for  Software  (DACS) 
contains  an  extensive  database  and  was  extremely  productive  in  obtaining 
information.  (See  references  section) 

c.  The  Institute  of  Electrical  &  Electronics  Engineering  (IEEE)  contains  an  extensive 
set  of  standards,  guidelines  and  draft  standards.  In  addition,  other  technical  IEEE 
publications  are  identified  that  served  as  an  extensive  source  of  information  (e.g.. 
Transactions  on  Software,  Computer  Magazine,  Software  Magazine,  Spectrum 
Magazine,  and  others).  The  Association  of  Computing  Machinery  (ACM)  provided 
additional  metrics  sources  and  related  information;  the  "Communications  of  the 
ACM"  publication  served  as  one  of  the  source  documents,  (see  references  section) 
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d.  Portland  State  University's  Metrics  database  and  Dr.  Wayne  Harrison  provided 
much  assistance  in  acquiring  information,  (see  references  section) 

e.  George  Mason  University's  Metrics  repository  and  Dr.  Ambrose  Goicoechea 
provided  direct  assistance  in  acquiring  metrics  information.  Professor  Goicoechea 
was  also  instrumental  in  acquiring  European  surveys  on  the  state-of-the-practice  in 
metrics,  (see  references  section) 

f.  The  NASA  Goddard  Space  Right  Center  Software  Engineering  Laboratory  and  the 
University  of  Maryland  have  an  extensive  amount  of  metrics  information.  Meetings 
have  been  held  with  NASA's  Frank  McGarry,  and  University  of  Maryland  (Dr.  M. 
Zelkowitz).  The  Research  Institute  in  Computer  and  Information  Sciences 
(RICIS),  University  of  Houston  (Dr.  C.  McKay)  was  also  contacted.  RICIS  is 
currently  engaged  in  Mission  and  Safety  Citical  software  initiatives  with  NASA's 
Johnson  Space  Center,  Houston  on  the  Space  Station  Software  Support 
Environment  (SSE).  Software  environment  concerns  for  the  SSE  are  similar  to 
those  that  are  expected  for  the  SDI's  (see  references  on  Basili  (Univ.  of  Md.), 
McKay) 

g.  The  Naval  Surface  We^>ons  Center  (NSWC)  at  Dahlgren,  Virginia  provided  details 
about  the  Statistical  Modeling  and  Estimation  of  ReliabUity  Functions  for  Software 
(SMERFS)  and  summary  data  on  Navy  software  support  tools. 

h.  The  Naval  Underwater  Systems  Command  (NUSC)  provided  descriptive  data 
about  their  management  methods  for  successful  software  metrics  application. 

i.  The  Headquarters  Command  at  Kiitland  AFB  described  their  enhanced  productivity 
model  "REVIC"  (outlined  in  Section  5). 


1.3  CROSS  REFERENCING  MODELS  BY  ATTRIBUTES  AND  PARAMETERS 


The  following  tables  (1.3-1  through  1.3-6)  summarize  the  metric  models  identified  in  the 
literature.  Each  table  groups  a  set  of  models  by  its  type  of  measurement  employed.  Currently  the 
groups  fall  into  six  measurement  categories.  The  categories  are:  1)  Time  Between  Failures,  2) 
Complexity,  3)  Failure  Count,  4)  Fault  Seeding,  5)  Input  Domain  Based,  and  6)  Productivity. 
Each  table  indicates  what  specific  software  attributes  are  addressed  by  each  model  in  the  table. 
This  identification  establishes  the  framework  for  the  mapping  of  so^are  metrics  to  software 
types,  processes,  and  domains  or  SDS  subfunctions.  Five  of  the  six  tables  use  the  same  attribute 
set;  while  the  table  on  productivity  uses  an  entirely  different  set.  The  models  dealing  with 
productivity,  while  concerned  with  performance  and  design  attributes,  are  focused  on  project 
management  issues  such  as  cost,  schedule,  and  the  overall  development  process.  For  this  effort, 
the  software  metric  models  addressing  the  nine  attributes  identified  in  will  be  covered.  In  addition 
to  identifying  metric  models,  this  section  also  contains  a  table  (1.3-7)  listing  the  most  commonly 
used  parameters  in  the  construction  of  software  metrics. 


The  majority  of  tables  use  nine  software  attributes.  These  are  the  attributes  identified  in 
Subtask  1,  TR-9033-1.  Thirteen  attributes  (factors)  were  originally  classified  by  Rome  Air 
Development  Center  (RADC);  however,  based  on  familiarity  with  SDI  component  software 
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requirements  several  attributes  were  combined  to  form  the  remaining  set  of  eight.  In  addition,  a 
ninth  attribute  was  proposed  in  TR -9033-1.  This  attribute  "throughput"  addresses  the  user 
concerns  of  computational  and  communication  throughputs.  The  following  alignment  shows  the 
current  nine  attributes  and  the  previously  defined  attributes  that  have  been  combined. 

CURRENT  ATTRIBUTE  OLD  ATTRIBUTE 


1. 

Reliability 

Reliability 

2. 

Survivability 

Survivability 

3. 

Integrity 

Integrity 

4. 

Efficiency 

Efficiency 

5. 

Maintainability 

Maintainability 

Correemess 

Verifiability 

Flexibility 

Expandability 

6. 

Usability 

Usability 

7. 

Portability 

Portability 

8. 

Reuse 

Reusability 

Reuse 

9. 

Throughput 

None 

1.3.1  Software  Quality  Attribute  Definitions 

The  following  definitions  are  included  here  for  completeness. 

Reliability.  The  probability  that  software  will  not  cause  the  failure  of  a  system  for  a  specified 
time  under  specified  conditions. 

Survivability.  The  built-in  capability  of  the  software  to  perform  its  required  function  when  a 
portion  of  the  system  is  inoperative. 

Integrity.  The  degree  to  which  the  software  controls  unauthorized  access  to,  or  modification 
of,  system  software  and  data. 

Efficiency.  The  degree  to  which  the  software  performs  its  intended  functions  with  minimum 
consumption  of  computer  time  and  storage  resources. 

Throughput.  The  degree  to  which  the  software  demonstrates  its  capability  to  process  data 
identified  as  computational  and/or  communications  related. 

Maintainability.  The  effort  required  to  locate  and  correct  an  error  in  the  software. 

Usability.  The  effort  required  to  learn  the  human  interface  with  the  software,  to  prepare 
input,  and  to  interpret  output  of  the  software. 

Portability.  The  effort  required  to  transfer  the  software  from  one  hardware  or  software 
environment  to  another. 
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Table  1 .3-1  Attributes  for  Time  Between  Failure  Models 


MODEL 


Time  Between  Failure 


Jelinsk'i/Moranda 
Schick/Wolverton  (Linear) 
Schick/Wolverlon  (Parabolic) 
Moranda  (Geometric  De-eut) 
Moranda  (Hybrid  Geomet  Poiss) 
Goel/Okumoto 
LittlewoodA/errall 
Llovd/Lioow 


Table  1 .3-2  Attributes  for  Complexity  Models 


ATTRIBUTES 

MODEL 

Performance  |  Design 

Reli  Surv  Intg  Effc  Thru  |  Usab  Main  Port  Reus 

Complexity 

1 

1 

Halstead 

McCabe 

Woodward  (Knot  Counts) 
Chen  (Nested  Oecisbn  Stmts) 


Benyon-Tinker 
Gib's  (Binary  Decision) 
Chapin's  Q 

Segment-Qbbal  Usage  Pair 
MyeTs  (McCabe  extension) 
Hansen's  (McCabe/Halstead) 
Oviedo's  (Data/Ctrt  Flows) 
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Table  1.3-3  AKributes  for  Failure  Count  Models 


ATTRIBUTES  I 

MODEL 

Performance 

1  Design 

Reli 

Surv  Intg  Effc  Thru 

1  Usab  Main  Port  Reus 

Failure  Count 

1 

1 

Goet/Okumoto 

X 

1 

Schneidewind 

X 

1 

Goel 

X 

1 

Musa 

X 

1 

Shooman 

X 

1 

Moranda 

X 

% 

1 

Jelinski/Moranda 

X 

% 

1 

Moranda 

X 

% 

1 

Schick/Wolverton 

X 

% 

1 

Goel/Okumoto  (Gen  Poisson) 

X 

% 

1 

Brooks/Motley 

X 

% 

1 

IBM  Poisson 

X 

% 

1 

%  Each  model  contains  a  different  representation  for  a  hazard 
function  that  may  be  adapable  to  support  sun/ivabiiity 

Table  1 .3-4  Attributes  for  Fault  Seeding  Models 


ATTRIBUTES 

MODEL 

Reli 

Performance  |  Design 

Surv  Intg  Effc  Thru  j  Usab  Main  Port  Reus 

1 

Fault  Seeding 

1 

1 

Mills 

X 

1 

Effc  •  Efficiency 
Intg  -  Integrity 
Main  >  Maintainability 
Port  -  Portability 
Reli  >  Reliability 


Reus  -  Reusabiitty 
Surv  >  SunrMbility 
Thai  ■  Throughput 
Usab  «  Usability 
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Table  1 .3-5  Attributes  for  Input  Domain  Based  Models 


MODEL 

Input  Domain  Based 

ATTRIBUTES 

Performance  |  Design 

Reli  Surv  Intg  Effc  Thnj  j  Usab  Main  Port  Reus 

1 

1 

_  -  _  -  _  _  _  1  - _ _  ^  - 

Nelson 

X 

—  1  ' 

1 

Ho 

X 

1 

Ramamoorthy/Bastan  i 

X 

1 

Effc  =  Efficiency 

Reus  >  Reusability 

Intg  =  Integrity 

Surv  «  SurvK/ibility 

Main  >  Maintainability 

Thru  ■  Throughput 

Port «  Portability 

Reli  o  Reliability 

Usab  «  Usability 

Table  1.3-6  Attributes  for  Productivity  Models 


MODEL 

ATTRIBUTES 

Productivity 

Size  Cost  Comp  Devp 

Software  Size 

X 

Personnel 

X 

Volatility 

X  .  X 

Resource  Utilization 
Complexity 

XXX 

Schedule  Progress 

XXX 

X 

Design  Progress 

X 

CSU  Development  Progress 

X  X 

X 

Testing  Progress 

X  X 

X 

Incremervtal  Release  Content 

XXX 

X 

Size  «  Size  of  the  Program 

Cost  -  Total  Cosyt 

Comp  -  Completion  Dale 

Devp  -  Effect  of  the  Development  Process 

I 
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Table  1 .3-7  Parameters  Associated  with  Metric  Classes 


METRIC  CLASSES 

T 

C 

F 

F  I 

PARAMETERS 

B 

P 

C 

S  D 

F 

X 

B 

Observed  time  between  failures 

X 

Calculated  hazard  function 

X 

Subjective  Random  Variable 

Cum  No.  of  Observed  failures 

X 

X 

X 

No.  of  faults  detected  during  a  time  period 

X 

X 

No.  of  faults  seeded  into  program 

X 

Failure  rate 

X 

X 

Expected  No.  of  faults  to  be  detected 

X 

X 

Amount  of  execution  time 

X 

No.  of  control  transfers 

X 

X 

No.  of  instructions  in  the  program 

X 

X 

Debugging  time  since  time  of  integration 

X 

No.  of  faults  corrected 

No.  of  entries/exits  per  module 

X 

X 

Software  science  measures 

X 

Design  structure 

X 

Data  flow  complexity 

X 

Requirements  traceability 

X 

Software  documentation 

X 

No.  of  operands  &  operators 

X 

"Binary"  decisions  in  program  logic 

X 

No.  of  external  interfaces 

X 

Segment  use  of  global  variables 

X 

No.  of  intersecting  control  statements 

X 

No.  of  compound  predicates 

Inputs  by  subdomains 

X 

X 

Metrics  Classes: 

TBF  -  Time  Between  Failures 

CPX  -  Complexity 

FC  -  Failure  Count 

FS  -  Fault  Seeding 

IDB  -  Input  Domain  Based 
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Reusability.  The  degree  to  which  the  software  can  be  used  in  multiple  applications. 

1.4  screemnt;  inappropriate  software  metrics 

A  review  of  the  metrics  contained  in  paragraph  1 .3  reveals  that  no  single  metric  is  capable  of 
supporting  decisions  across  an  entire  software  domain  as  defined  in  Subtask  1,  TR -9033-1.  Each 
metric  supports  measurements  in  one  or  two  software  attributes;  e.g..  Reliability,  Maintainability, 
etc.  The  implication  here,  is  that  perhaps  several  metrics  will  need  to  be  combined  to  form  a 
composite  model  capable  of  supporting  decisions  on  the  36  SDS  subfunctions  or  the  defined 
software  domains.  The  analysis  concludes  that  no  available  metric  should  be  dropped  from 
consideration. 


1.5  MAPPINf;  SOFTWARE  METRICS  TO  TYPES  AND  PROCESSES 

Each  of  the  software  metrics  classes  presented  is  mapped  into  its  applicable  major  function 
and  subfunction,  using  the  attribute  rankings  developed  in  TR-9033-1 . 

1.5.1  Ranking  Attributes  by  Software/Process  Types 

First,  the  characteristics  of  each  subfunction  were  studied  to  determine  the  relative  weights  of 
the  attributes.  This  analysis  was  performed  in  Subtask  1  and  is  explained  in  SDIO  Task  33,  TR- 
9033-1  SDS  Software  Measurement  Requirements.  Para.  2.3  "Characteristics  of  Software."  The 
following  characteristics  were  postulated;  a)  Criticality,  b)  Embedded  vs.  general  purpose,  c) 
Space/Ground  based,  d)  Life  cycle,  e)  Algorithmic  Content,  f)  Size,  g)  Risk,  and  H)  Intended  use. 

Then,  in  TR-9033-1,  Para.  2.4,  the  characteristics  of  each  subfunction  were  used  to 
determine  the  attribute  rankings  for  each  SDS  Subfunction. 

Table  1.5-1  summarizes  the  subfunction  attribute  rankings,  concentrating  upon  the  "high" 
and  "medium"  rankings;  the  remaining  subfiinction-attribute  correlations  were  all  "low." 

1.5.2  Subfunction  Metrics  Applicabilities 

The  applicability  of  the  metrics  classes  and  models  to  each  of  the  quality  attributes  was 
presented  in  Para.  1 .3  and  Tables  1.3-1  through  1.3-6.  These  applicabilities  are  used  as  a  basis  to 
derive  the  subfiinction  metrics  applicabilities. 

Table  1.5-2  presents  the  extrapolations  to  metrics  classes  from  the  anribute.and  applicability 
rankings,  and  Table  1.5-3  presents  Ae  derivation  mles  used. 

1.5.3  Development  Process  Applicability 

Prior  to  the  analytical  evaluation  of  the  metrics  in  regard  to  the  subfunction  requirements, 
(presented  in  Para.  2)  we  need  to  consider  how  the  development  process  can  affect  metrics 
applicabilities.  The  possibility  of  employing  off-the-shelf  and  preused  software  componenets 
presents  a  different  software  metrics  envirorunent  than  the  single-vendor  prototype  concept,  and 
the  traditional  requirements-based,  multi-phased  development. 


TASc! 

rmt 

•PANT* 
Mtl  1 

•RC  ' 

om*  1 

1  WTIMW  STA  TIAM  i 

1  uuAiitnmTotoc  } 

1-16 


THE  ANALYTIC  SCIENCES  CORPORATION 


Table  1 .5-1 .  Software  Functions  with  Ranked  Attributes _ (from  TR-9033-1) 


Major  Function 

No.  Title 

Subfunction 

No.  Title 

High/Med  Applicable  Attributes 

Reli  Surv  Intg  Thru  Usab  Main  Port 

Reus  Effc 

1  Detect 

1  Plumes 

H 

H 

H 

H 

M 

2  Coldtxxjies 

H 

H 

H 

H 

H 

3  RF 

H 

H 

H 

H 

M 

M 

M 

2  Identify 

4  Resolve  Obj. 

H 

H 

H 

M 

M 

5  Discr 

H 

H 

H 

M 

M 

6  Assess  Kills 

H 

H 

H 

M 

M 

M 

3  Track 

7  Correlate 

H 

H 

H 

H 

M 

8  Initiate 

H 

H 

H 

H 

M 

9  Estimate 

H 

H 

H 

H 

M 

M 

10  Predict  l&l 

H 

H 

H 

M 

M 

4  Communicate 

11  Interplatform 

H 

H 

H 

H 

M 

H 

12  Ground-Space 

H 

H 

H 

M 

M 

M 

M 

13  Ground 

M 

M 

M 

M 

M 

H 

M 

H 

5  Assess 

14  Threat 

H 

H 

H 

15  SDS 

H 

H 

H 

6  Wpn  Contrl 

16  SBI  Asgn&Ctrl 

H 

H 

H 

H 

M 

17GBI  Asgn&Ctrl 

H 

H 

H 

M 

H 

M 

18  SBI  Guide&Ctri 

H 

H 

H 

H 

H 

19  GBI  Guide&Ctrl 

H 

H 

H 

M 

H 

M 

7  Platfm  Mgt 

20  Cmd  Env  Ctrl 

H 

M 

H 

M 

M 

H 

M 

M 

21  Ctrl  Onbd  Env 

H 

H 

H 

H 

M 

22  Cmd  Attit&Pos 

H 

M 

H 

M 

M 

H 

M 

M 

23  Ctrl  Attit&Pos 

H 

H 

H 

H 

M 

24  Sense  Status 

H 

H 

H 

M 

M 

25  Assess  Status 

H 

M 

H 

M 

M 

H 

M 

M 

26  Cmd  Reconfig 

H 

M 

H 

M 

M 

H 

M 

M 

27  Reconfigure 

H 

H 

H 

H 

M 

8  Spprt  Dev 

28  Tools 

M 

M 

M 

M 

M 

M 

M 

9  Simulate 

29HWIL 

M 

H 

H 

M 

M 

M 

M 

30  Demonstration 

M 

H 

M 

M 

M 

M 

M 

M 

10  Spprt  Acqu 

31  Developer  Test 

M 

M 

M 

M 

H 

H 

H 

H 

M 

32  Developer  Env 

H 

M 

H 

M 

H 

M 

M 

33  Factory  Test 

M 

M 

M 

H 

H 

H 

H 

34  Acceptance  Test 

M 

M 

M 

M 

M 

M 

1 1  Spprt  Mgt 

35  MIS  DB  Maim 

M 

M 

H 

M 

H 

H 

H 

H 

M 

36  Track  Mgt  Info 

M 

H 

H 

H 

H 

H 

M 

ATTRIBUTES;  Reli  >  Reliability  Main  -  Maintainability 

Surv  -  Survivability  Usab  -  Usability 

Intg  >  Int^rity  Reus  >  Reusability 

Effc  -  Efficiericy  Port  -  Portability  Thru «  Throughput 
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Table  1 .5-3;  Metrics  Ranking  Derivation  Rules 


ATTRIBUTE  RANKING 

METRIC  CLASS/MODEL  RANKING 

<RELIABILITY  *RELr> 

**> 

<TIME  BETWEEN  FAILURES  TBP> 
&<COMPLEXITY  -CPX"> 
&<FAILURE  COUNT  TC"> 
&<FAULT  SEEDING  *FS"> 
&<INPUT  DOMAIN  BASED  ■|DB"> 

<SURVIVABILITY  "SURV''> 

KS> 

<TBF>  &  <FC> 

HIGH  "H" 

H 

MEDIUM  "M" 

**> 

M 

H&M 

==> 

H 

H&H 

*»> 

H 

M&M 

M 
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Two  classes  of  metrics  assume  access  to  source  code:  Complexity  Models  (Halstead, 
McCabe,  Woodward,  etc.)  and  the  Mills  Fault-Seeding  Model,  which  also  requires  the  alteration  of 
the  source  or  the  software  adaptation  data  (parameters,  etc.).  The  failure-sensitive  (TBF  &  FC) 
and  input  domain  (IDB)  and  many  productivity  metrics  models  are  independent  of  the  source  code, 
and  do  appear  applicable  to  hybrid  development  processes. 

Other  developmental  process  implications  are  discussed  next  in  regard  to  the  software 
development  life  cycle  (SDLC). 

1.6  LIFECYCLE 

1. 6. 1  Life  Cycle  Models 

This  section  on  life  cycles  has  a  fourfold  purpose: 

a.  To  establish  a  reference  life  cycle  to  support  the  categorization  of  metrics,  and  the 
particular  phase  with  which  they  are  associated  or  used; 

b.  To  establish  a  foundation  and  reference  point  from  which  subsequent  and  different 
life  cycle  forms  can  be  derived  or  related; 

c.  To  establish  a  reference  model  that  can  be  used  in  the  identification  of  new  metric 
forms  and  types,  and  the  phase  in  which  they  will  be  employed; 

d.  To  provide  the  basis  for  establishing  new  metrics  methodologies  and  usage 
emphasis. 

To  support  the  categorization  of  metrics,  a  reference  or  standard  life  cycle  required 
identification  together  with  its  identified  phases.  The  software  life  cycle  used  is  that  defined  in 
[MIL-STD-2167A].  A  number  of  other  life  cycle  variations  were  found  (RADC878A,  IEEE  Std. 
P982.2/D6,  [BOEH81])  and  examined.  Life  cycles  were  found  to  vary  in  the  number  and 
definition  of  phases.  However,  these  differences  for  the  most  part  were  found  to  be 
organizational,  with  the  content  essentially  the  same  as  Tables  1.6-1,2&3  illustrate.  Life  cycle 
variations  were  minor,  and  all  of  these  models  can  be  mapped  into  each  other  with  relative  ease,  as 
well  as  the  waterfall  model  of  TR-9033-1. 

Newer  life  cycle  models  (i.e..  Spiral,  technology,  iterative)  that  more  appropriately  support 
complex  development  activities  such  as  prototyping  and  reusability  paradigms  were  not  considered 
for  evaluation.  The  newer  models  have  not  been  used  to  the  extent  that  the  classical  waterfall 
model  has,  and  thus  not  much  experience  and  data  are  available.  However,  their  iterative  nature, 
("build  a  little,  test  a  little,  correct/adjust,  repeat  the  process  again"),  coupled  with  the  formality  of 
some  of  the  prototypes  used  in  this  iterative  process,  is  more  appropriately  suited  to  modem  day 
developments.  Large  and  complex  system  developments  that  must  endure  changing  requirements, 
as  well  as  undergo  evolutionary  or  incremental  changes  must  be  supported  by  iterative  processes 
that  provide  the  ability  to  revisit  and  reexamine  design  incursions  and  baseline  changes.  It  is, 
therefore,  recommended  that  a  more  indepth  examination  of  iterative  life  cycle  models  be  made  in 
the  context  of  this  report. 
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Items  a.  and  d.  are  considered  within  the  scope  of  this  task,  items  b.  and  c.  are  outside  the 
scope  of  this  effort.  The  basis  for  the  scope  statements  is  the  pragmatic  task  emphasis  of 
identifying  existing  and  available  software  and  system  measurement,  and  variably  reliable 
predictors  of  high  utilitarian  value  to  managers  and  technical  personnel. 

A  few  observations  about  the  referenced  software  development  life  cycles  contained  in 
Tables  1. 6- 1,2,3: 

User  and  developer  bidirectional  communications  and  information  flow  are  not  well 
supported  (e.g.,  human  engineering  and  software  engineering).  This  is  indicative 
of  the  lack  of  feedback  paths  within  the  life  cycles  required  to  resolve  deficiencies, 
issues  and  problems  as  Aey  arise; 

Events  and  activities  are  sequential  in  nature,  with  very  little  support  for  iterative 
processes.  The  2167A  life  cycles  support  prototyping  and  design  reusability. 
Iterative  processes  are  required  to  tune  or  refine  prototypes  as  design  information 
evolves  and  is  elaborated  upon; 

Resulting  products  from  such  are  limited  and  constrained.  The  varying  degrees  of 
prototype  and  reuse  formalisms  will  require  tailoring  of  existing  specifications,  as 
well  as  require  new  ones.  Additionally,  associated  design  reviews  will  also  require 
customizing  (e.g.,  reuse  preliminary  design  review,  prototype  specification  design 
review). 

More  sophisticated  configuration,  delivery  and  requirements  packaging,  such  as  incremental 
development,  incremental  deployment,  evolutionary  development,  transactional  threading,  and 
technology  insertion  have  given  rise  to  newer,  less  well  defined  models.  Models  to  support 
various  advanced  life  cycle  requirements  have  had  slow  acceptance  (formal  prototype,  executable 
prototype,  spiral,  automation-based  model). 

The  applicability  of  each  metric  to  the  life  cycle  phases  must  be  clearly  identified.  This  is 
essential  in  planning  for  effective  visability  and  predictability  in  advance  of  development.  A  metric 
used  to  assess  code  structure  is  one  that  is  invoked  late  in  the  development  life  cycle  where  much 
time  and  resources  have  already  been  consumed  (post-facto).  A  metric  or  predictor  used  in  the 
early  or  initial  life  cycle  phases  (i.e.,  requirements/design  synthesis)  can  provide  anticipatory  or 
predictive  insight  before  long  term  resources  are  committed,  where  it  is  much  more  cost-effective 
[BOEH81]  to  make  design  tradeoffs  and  correct  problems. 
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TABLE  I.6-I 

COMPARINC,  MIL-STD.2I67A  &  RADC-TR.87.171 


MIL-STD-2167A 
No.  Phase 

1  S/W  Reqts.  Analysis 

2  Preliniinary  Design 

3  Detailed  Design 

4  Coding  &  Unit  Test 

5  CSC  Integration  &  Test 

6  CSCI  Testing 

7  System  Integ.  &  Testing 

8  C^r.  Test  &  Evaluation 

9  Deployment  &  Distribution 


RADC-TR-87-171 

No.  Phase 

1  S/W  Reqts.  Analysis 

2  Preliminary  &  Detailed  Design 

3  Coding  &  Unit  Test 

4  CSC  Integration  &  Test 

5  CSCI  Testing 

6  System  Integ.  &  Testing 

7  C^r.  Test  &  Evaluation 

8  Production  &  Deployment 


TABLE  1.6-2 


COMPARING  1V1IL-STD.2167A  &  IBOEHSll  WATERFALL 


MIL-STD-2167A 
^  Pha§^ 

1  S/W  Reqts.  Analysis 

2  Prelimiiuuy  Design 

3  Detailed  Design 

4  Coding  &  Unit  Test 

5  CSC  Integration  &  Test 

6  CSCI  Testing 

7  System  Integ.  &  Testing 

8  C^r.  Test  Sc  Evaluation 

9  Deployment  &  Distribution 


[BOEH8 1  ]  WATERFALL  LIFE  CYCLE 
No.  Phase 

1  Software  Reqts./Validation 

2  Product  Design/Verification 

3  Detailed  Design/Verification 

4  Code/Unit  Test 

5  Integ./Product  Verification 

6  Implementation  System  Test 

7  Op's  &  Maint./Revalidation 
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TABLE  1.6-3 


COMPARINC.  RAD(:.TR.87.17I 


RADC-87  UFE  CYCLE 
No.  Phase 

1  SAV  Reqts,  Analysis 

2  Preliminary  and  Detailed 

Design 

3  Coding  &  Unit  Test 

4  CSC  Integration  &  Test 

5  CSCI  Testing 

6  System  Integ.  &  Testing 

7  C^r.  Test  &  Evaluation 

8  Production  &  Deployment 


&  fBOEHHIl  WATERFALL  PHASES 


[BOEH81]  WATERFALL  UFE  CYCLE 
No.  Phase 

1  Software  Reqts. /Validation 

2  Product  Design/Verification 

3  Detailed  Desigrt/Verification 

4  Code/Unit  Test 

5  Integ./Product  Verification 

6  Implementation  System  Test 

7  Op's  &  Maint./Revalidation 


As  indicated  in  Subtask  1,  TR-9033-1,  approximately  70%  of  all  "software  eirors"  actually 
occur  early  in  the  development  cycle.  This  gives  great  importance  to  identifying  and  implementing 
those  metrics  which  can  be  applied  in  the  early  design  phases.  An  examination  of  a  well  defined 
characteristic  life  cycle  model  (e.g.,  RADC’s)  as  presented  in  Table  1.6-4,  with  its  associated 
reliability  measure  model  (Table  1.6-5)  reveals  the  following: 

-  The  identification  of  a  prototyping  phase  and  an  extensible  reusability  dependent 
life  cycle  (i.e.,  reusable  architecture,  design,  specifications  and  code)  requires  the 
identification  of  new  metrics  or  redefinition  of  others.  Thus,  the  entry  for  reuse 
metrics  (Table  1.6-4)  would  also  appear  in  the  preliminary  and  detailed  design 
phase;  while  a  new  one  would  be  entered  in  this  same  phase  for  prototyping. 

-  Secondly,  the  associated  reliability  measurement  model  of  Table  1.6-5  would 
require  similar  adjustments  in  its  metrics  formula  representation. 

-  Thirdly,  a  shift  in  emphasis  would  occur,  from  estimation  to  predictive  metrics,  if 
emphasis  were  placed  on  prototyping  a  system  in  order  to  obtain  early  visibility 
into  the  design  and  ferret  out  initid  design  issues. 

The  impact  on  the  reliability  model  predictive  and  estimation  metrics  (Table  1.6-5) 
relationships  (see  reference  for  formuli  details)  can  be  altered  significantly  if  new  terms,  such  as 
NR  and  NP  are  added  to  this  established  model. 

Furthermore,  in  both  the  RADC  and  IEEE  models,  predictive  metrics  (category  1)  used  in  the 
early  life  cycle  phases  are  the  ones  where  the  least  amount  of  information  is  available  and  where 
fortrud  relationships  are  least  understood. 
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TABLE  1.6-5 

SOFTWARE  RELIABILITY  MEASUREMENT  MODEL 
RADC-TR-87.17I  PREDICTIVE  AND  ESTIMATION  METRICS 

PART  1:  PREDICTIVE  METRICS 


Application  Type  A 

Development  Environment  D 

Software  Characteristics  S 

Requirements  and  Design  Representation  S 1 

Anomaly  Management  SA 

Traceability  ST 

Quality  Review  Results  SQ 

Software  Implementation  S2 

Language  Type  SL 

Program  Size  SS 

Modularity  SM 

Extent  of  Reuse  S  U 

Complexity  SX 

Standards  Review  Results  SR 


Rp  =  A  X  D  X  S  where 


S  =S1  xS2 

51  =SAxSTxSQ 

52  =  SLxSS  X  SM  X  SU  X  SX  X  SR 


PART  2:  ESTIMATION  METRICS 


Failure  Rate  During  Testing 
Test  Environment 
Test  Effort 
Test  Methodology 
Test  Coverage 
Operating  Environment 
WoiUoad 
Ii^ut  Variability 


RE  =  F  X  T,  during  testing  where 
RE  =  F  X  E,  during  OT&E  where 


T  =  TE  X  TM  X  TC  and 
E  =  EWxEV 


F 

T 


E 
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Focusing  on  MIL-STD-2167A  thus  establishes  the  need  for  the  definition  of  a  more 
extensive  and  formal  life  cycle  that  can  address  the  activities  and  processes  for  prototyping  and 
reusability  among  others. 

1.6.2  Life  Cycle  Experience  Classification 

Table  1.6-6  represents  an  experience  classification  matrix  of  measures  and  the  life  cycle 
phase  in  which  their  use  is  appropriate.  The  table  divides  the  measures  into  three  categories: 

Category  1  -  identifies  measures  that  have  been  formalized  (e.g.,  by  a  mathematical 
foundation) 

and  have  limited  or  insufficient  operational  validation 

Category  2  -  identifies  measures  that  have  been  formalized  and  have  a  moderate 
experience  validation 

Category  3  -  identifies  measures  that  have  been  formalized  and  have  an  extensive 
experience  basis. 

The  table  serves  to  identify  measures  (category  1)  that  may  have  a  higher  utilitarian  value 
than  the  currently  utilized  metrics  of  category  3  [IEEE  P982]  that  are  well  understood  and  industry 
recognized  (e.g.,  McCabe,  Halstead).  It  is  interesting  to  note  that  category  3  measures  fall  into  the 
later  phases  of  the  life  cycle  further  supporting  the  claim  that  they  are  post-facto  measures  (i.e., 
occur  from  the  coding  phase  to  the  operations  and  maintenance  phase)  for  the  most  part  where 
significant  resources  have  been  consumed  and  committed.  While  category  3  measures  represent 
approximately  20%  of  the  total  and  are  represented  by  the  majority  of  tools  that  exist  in  the 
marketplace,  category  1  measures  account  for  approximately  50%  and  ate  not  supported  by  many 
tools  and  environments.  The  remainder  fall  into  category  2  (approximately  30%). 

A  contributing  factor  to  the  unavailability  of  metrics  information  rigor  in  the  early  phases  of 
the  life  cycle  is  the  fact  that  use  of  formal  notation  to  support  requirements  and  design  synthesis  is 
not  available  or  utilized  within  the  industry  for  the  most  part.  The  sparse  use  of  formal  notation 
within  the  life  cycle  reveals  a  serious  technology  and  communications  gap  between  the  systems  and 
software  engineering  communities.  Use  of  a  formal  syntax  (e.g.,  BNF  notation)  or  system  design 
language,  with  the  same  level  of  formalism  as  those  that  exist  in  the  irrq)lementation  phase  of  the 
life  cycle  (i.e.,  programming  language)  has  the  benefit  of  being  machine  processable  and 
analyzable. 

The  coexistence  of  both  a  formal  system  design  and  a  programming  language  provides  the 
basis  for  applying  complexity,  efficiency,  reliability,  etc.,  measures  in  a  more  consistent  and 
predictive  manner.  The  existence  of  a  formal  system  design  language  allows  an  early  assessment 
of  design  integrity  and  completeness  that  can  in  turn  Ire  used  to  predict  code  and  structural 
complexity,  and  ease  of  integration.  Formal  notation  would  provide  metrics  measurements  and 
models  with  a  mote  extensive  and  effective  range  of  application. 
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Parti;  LIGHT  EXPERIENCE  METRICS  BY  LIFE  CYCLE  PHASE 


Cateqorv  1  Metric 


um.  t-aiiure  Krotiie 
Cum.  Failure  Profile 
Functional/  Modular  Test  Coverage 
Defect  Indices 
Errors  Distributions 
S/W  Maturity  Index 
Number  Entry/Exits  Per  Module 
Graph-Theoretic  Cmplexity  for  Arch. 
Design  Structure 
Software  Purity  Level 
Requirements  Complicance 
Data  or  Info.  Row  Complexity 
Residual  Faults  Count 
Testing  Sufficiency 
Required  Software  Reliability 
Software  Release  Readiness 
Test  Accuracy 
Indep.  Process  Reliability 
Combined  HW/SW  Operational  Reli. 


Cone  Rqmts 


X 


Implem  Test 


x 


Intg/Tes  O&M 


Part  2;  MODERATE  EXPERIENCE  METRICS  BY  LIFE  CYCLE  PHASE 


Cateqorv  2  Metric  Cone  Rqmts  Des  Implem  Test  Intq/Tes  O&M 


Cause  &  Effect  Graphing 
Manhours  per  Major  Defect  Detected 
Number  of  Conflicting  Rqmts 
Minimal  Unit  Test  Case  Determination 
Run  Reliability 

Estimated  Number  of  Faults  Remaining 

Test  Coverage 

Reliability  Growth  Function 

Software  Documentation  &  Source  Listing 

Completeness 

Svstem  Performance  Reliabilit 


Pan  3:  GREATER  EXPERIENCE  METRICS  BY  LIFE  CYCLE  PHASE 


etric 


ensity 

Requirements  Traceability 

Software  Science  Measures 

Cyclomatic  Complexity 

Mean  Time  To  Discover  The  Next  K  Faults 

Failure  Analysis  Using  Elapsed  Time 

Mean  Time  To  Failure 

Failure  Rate 


em  les 


Table  1.6-6:  SOFTWARE  METRICS  EXPERIENCE  CLASSIFICATION 


TSC 
SPAKTA 
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BNC 
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1.6.3  Summary 

Paragraphs  1.6.1  and  1.6.2,  together  with  the  other  sections  of  this  report,  are  intended  to 
provide  the  basis  for  the  following; 

a.  Much  more  metric  information  and  formalism  is  concentrated  on  the  later  phases  of 
the  life  cycle  (i.e.,  from  coding/implementation  to  the  operations  and  maintenance 
phases).  The  reference  section  provides  additional  strong  support  for  the  statement. 

b .  The  cost  to  correct  errors  or  deficiencies  in  the  later  phases  increases  by  at  least  one 
to  two  orders  of  magnitude.  This  is  attributable  to  the  consumption  of  resources 
and  labor  intensive  activities  that  have  occurred  by  the  time  that  implementation  or 
coding  is  reached  [BOEH81]. 

c.  The  coding  and  related  phases  (e.g.,  CSC  phase)  of  the  life  cycle  are  the  least 
significant  cost  drivers.  It  is  recognized,  thus  sjjecific  references  are  not  required, 
that  maintenance  is  the  costliest  phase,  as  a  result  of  redesign,  requirements 
changes,  poor  design  visibility  and  spiecifications. 

d .  Current  development  efforts  that  have  focused  on  the  early  phases  of  the  life  cycle, 
predictive  metrics  and  the  early  detection  of  problems  have  had  very  good 
productivity  (cost  and  schedule)  and  have  produced  quality  software  with  fewer 
catastrophic  errors.  IBM's  Space  Shuttle  Program,  NUSC's  Submarine  Combat 
System,  Unisys  Trident  Submarine  Navigation  Program,  and  Teledyne  Brown's 
SDI  SIE  and  N-SITE  programs  are  examples  of  such.  'Though  these  efforts  are 
few  in  number,  a  consistent  theme  is  emerging. 

Indications  are  that  focusing  on  predictive/early  life  cycle  use  of  metrics  and  focusing  on  the  early 
life  cycle  phases  themselves,  will  provide  a  better  quality  and  cost-effective  design.  Yet  predictive 
metrics  and  the  measures  of  category  1  (Table  1 .6-6)  are  the  ones  with  the  least  experience  factors 
and  understanding. 
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2.  ANALYTIC  EVALUATION  OF  SOFTWARE  METRICS 


2.1  FEATURE  ANALYSIS 

As  outlined  in  Subtask  1,  TR-9033-1,  several  software  domains  can  be  constructed  using 
combinations  of  characteristics  and  factors  applied  to  36  SDS  subfunctions.  An  assessment  of 
how  each  applicable  software  metric  can  be  applied  within  each  SDS  subfunction  will  be 
presented.  Each  table  in  Appendix  B  identifies  the  SDS  subfunction  and  the  nine  software 
attributes  along  with  their  relative  rankings  as  presented  in  Subtask  1,  TR-9033-1,  for  that 
subfiinction.  Each  relative  ranking  is  assigned  a  score.  The  scores  are  arbitrarily  assigned  a  value 
from  1,  for  low,  to  5,  for  high. 

A  vector  is  created,  for  each  identified  metric,  using  the  scores  assigned  from  the  relative 
rankings  given  in  TR-9033-1.  For  example,  the  first  subfunction  identified  is  "Detect  Plumes". 
Associated  with  that  subfunction  are  the  following  attributes  with  their  relative  rankings.  Beside 
each  relative  ranking  is  its  associated  score. 


Quality  Attributes 

Relative  Rankings 

Score 

Reliability 

High 

5 

Survivability 

High 

5 

Integrity 

High 

5 

Efficiency 

Moderate 

3 

Throughput 

High 

5 

Usability 

Low 

1 

Maintainability 

Low 

1 

Portability 

Low 

1 

Reuse 

Low 

2 

Taking  the  McCabe  metric  from  Table  1.3-2,  note  that  this  metric  is  used  in  measuring  reliability 
and  maintainability.  For  Detecting  Plumes,  the  need  for  reliability  as  defined  in  TR-9033-1  is 
high,  while  the  need  for  maintainability  is  low.  Since  these  are  the  only  two  attributes  measured  by 
this  metric  the  following  vector  is  created; 


Detect  Plumes 
TR-9033-1  Rankings 


H 

H 

H 

H 

L 

L 

L 

L 

M 

BsU 

Surv 

Intg 

Thru 

Usab 

Main 

Port 

Reus 

Effc 

McCabe 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Recognize  that  each  subfunction  has  a  different  set  of  attribute  rankings  and  therefore  creates  a 
different  vector  set. 
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Using  the  McCabe  metric  for  another  SDS  subfunction,  e.g.,  "Command  Environment 
Control",  the  vector  is  different  and  it  appears  more  relevant  to  this  subfunction  than  Detecting 
Plumes. 


Command  Environment  Control 


TR-9033-1  Rankings 


H 

M 

H 

M 

M 

H 

M 

L 

M 

Reli 

Surv 

Intg 

Thru 

Usab 

Main 

Port 

Reus 

Effc 

McCabe 

5 

0 

0 

0 

0 

5 

0 

0 

0 

The  McCabe  metric,  though  still  deficient  in  six  of  the  eight  attributes,  matches  the 
requirements  of  the  Command  Environment  Control  subfunction  better  since  it  would  be  able  to 
measure  2  of  the  3  attributes  designated  as  a  high  need. 

The  reader  should  note  the  large  number  of  zeroes  included  in  the  vectors  shown  in 
Appendix  B.  This  indicates  that  the  established  metrics  do  not  support  many  of  the  SDS  required 
attributes.  It  appears  that  much  work  still  needs  to  be  accomplished  in  creating  metrics  that  address 
those  attributes.  One  method  of  attacking  this  deficiency  would  be  to  use  an  emerging  class  of 
metrics  designated  as  multi-attribute  metrics.  Some  pioneering  work  is  currently  on-going  at 
George  Mason  University  and  other  academic  institutions  around  the  country.  In  fact,  the  National 
Science  Foundation  sponsors  a  Special  Interest  Group  on  Multi-Criteria  Decision  Making  (MCDM) 
which  meets  periodically  to  discuss  the  state-of-the-art  in  software  measurement. 

An  alternative  approach  would  be  to  investigate  the  feasibility  of  creating  metrics  for  those 
attributes  not  currently  addressed.  After  those  metrics  are  created,  a  composite  (or  collective)  set  of 
metrics  could  be  applied  to  each  SDS  subfiinction.  The  greatest  payback  would  be  achieved  by 
af^lying  those  composite  or  multi-attribute  metrics  that  address  the  attributes  with  high  or  moderate 
relative  ranking.  If  funds  still  remained  in  the  quality  assurance  purse  after  covering  higher  level 
attributes,  then  the  anributes  designated  as  low  ranking  could  be  addressed. 

Caution  must  be  exercised  in  inteipreting  the  values  (scores)  in  the  created  vectors.  No  one 
should  make  a  judgement  that  one  metric  is  "better"  than  another  because  a  vector  value  is  larger, 
e.g.,  5  vs.  1.  Remember  the  scores  are  highly  dependent  on  the  relative  ranking  derived  in  TR- 
9033-1  for  each  SDS  subfunction.  Even  the  same  metric  can  have  several  different  vectors, 
depending  again  on  the  SDS  subfunction  being  evaluated.  The  assessment  that  can  be  made  for  a 
vector  with  higher  scores  is  that  the  triplication  of  that  metric  to  its  specific  SDS  subfunction  will 
provide  infomtation  about  an  attribute  assessed,  in  TR-9033-1,  to  be  more  critical.  This  report 
does  not  claim  to  rank  the  metrics.  It  attempts  to  show  where  specific  metrics  can  best  be  applied 
effectively  across  all  36  SDS  subfunctions. 

The  well-defined  metrics  which  were  analyzed  in  this  section  are  generally  {qiplied  late  in  the 
software  development  life  cycle  (SDLC).  They  are  normally  classified  as  estimation  type  metrics 
rather  than  prediaive.  The  current  trend  is  to  develop  and  apply  predictive  metrics  early  in  the 
SDLC  to  aid  not  only  program  managers,  but  also  development  personnel  in  identifying 


TAScf 

•PAKT* 

Ml 

.  me  I 

DS*  : 

1  IWMTKMWtrrATUM  ( 
1  flOMUnOTOM  1 

2-2 


THE  ANALYTIC  SCIENCES  CORPORATION 


unfavorable  trends  early  in  development  where  corrective  action  can  be  taken  with  minimum  cost 
and  schedule  impacts.  As  previously  presented  in  Table  1.6-4,  the  more  research  and  development 
done  to  inspire  predictive  metrics  for  the  preliminary  and  detailed  design  phase  of  the  life  cycle,  the 
better  the  control  will  be  on  future  developmental  efforts. 
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3. 


EXPERIENCE  ASSESSMENT 


A  number  of  major  programs  were  identitied  as  potential  sources  that  would  provide  field 
experience  and  relevant  insight  into  software  metrics.  Several  programs  were  eliminated  initially 
due  to  their  states  of  maturity  or  information  availability.  NASA's  Space  Station  initiatives  are  in 
their  initial  phases,  thus  a  shift  to  other  programs  with  a  greater  experiential  base  was  merited 
(e.g.,  NASA^BM  Shuttle  program).  However,  that  is  not  to  say  that  the  Space  Station  program 
does  not  have  a  quality  or  metrics  program.  Some  of  the  RICIS  literature  and  effort  clearly  focuses 
on  mission  and  safety  critical  software  and  related  issues  for  the  Space  Station. 

The  Joint  Tactical  Fusion  program  has  a  viable  metrics  initiative.  Though  information 
requested  is  forthcoming,  the  classified  nature  of  the  program  precludes  the  screening  of  relevant 
metrics  information  in  time  for  this  report.  However,  review  of  models  used  and  metrics  initiatives 
are  consistent  with  and  support  conclusions  reached  in  this  report. 

Formally  instituted  metrics  programs  on  major  systems  acquisition  are  not  readily 
identifiable,  with  the  exception  of  those  noted  in  the  foUowing  sections.  For  the  most  part,  metrics 
efforts  are  contained  within  larger  quality  assurance  programs  or  as  smaller  independent  research 
initiatives  within  DoD.  However,  where  metrics  programs  have  been  highlighted  as  part  of  major 
program  acquisitions,  much  success  has  followed  those  developments/acquisitions. 

3.1  THE  NAVAL  UNDERWATER  SYSTEMS  COMMAND  (NUSC) 

The  NUSC  facility  at  Newport,  Rhode  Island,  has  been  using  a  suite  of  software  metrics  to 
manage  submarine  combat  systems  acquisitions  for  the  last  four  years.  Dr.  John  Short,  Dept. 
Manager,  was  instrumental  in  setting  the  management  guidance  for  a  cohesive  method  of  contractor 
oversight  using  software  metrics  throughout  the  life  cycles  of  each  of  the  emerging  developments. 

The  management  concept  at  work  is  risk  management:  to  manage  project  and  resource  risks 
by  assessments,  predictions,  validations  and  risk  mitigations.  The  process  starts  with  RFP 
development  and  proceeds  apace  of  the  life  cycles.  Key  players  within  NUSC  and  the  contractors 
are  given  training  and  briefings  by  the  NUSC  software  metrics  staff. 

The  NUSC  database  and  experience  is  focused  on  the  Navy  Submarine  Combat  System 
Applied  Software  Metrics  Program.  NUSC  has  approximately  10  million  lines  of  source  code 
(MSLOC)  in  process  with  projects  varying  between  0.5  and  4.0  MSLOC. 

The  NUSC  software  metrics  program  applications  have  initially  centered  on  CMS-2  language 
applications,  with  two  recent  exceptions,  the  CCS  MK  2  and  AN/BSY-2  programs  using  Ada  and 
^^L-STD-2167.  Key  tools  used  in  the  NUSC  program  are  LOTUS  1-2-3  as  the  spreadsheet  for 
information  collection  of  such 

items  as  faults,  problem  trouble  reports  (PTR),  and  PTR  testing,  supported  by  an  ORACLE 
database;  at  least  4  development  models  (SLIM,  COCOMO,  SECOMO,  SASET);  2  reliability 
models  (MUSA,  DUANE);  a  complexity  analyzer  and  code  analyzer  (AdaMat);  a  variety  of  PC- 
based  management  tools,  and  other  non-  metrics  tools.  In  summary,  the  Navy's  program  is 
focused  on  applied,  tailorable  and  practical  applications  oriented  metrics.  Although  the 
implementation  j^ase  is  monitored  and  evaluated,  initial  life  cycle  phases  ate  strongly  emphasized. 
The  collection  and  analysis  of  problem  trouble  reports,  together  with  statistical  evduation  of  them 
(rate  of  occurrence,  rate  of  resolution,  resolution  to  occurrence  correlation)  plays  a  critical  role  in 
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predicting  software  quality  for  NUSC.  The  effort  has  potentially  significant  relevance  to  the  SDI 
initiative  and  parts  and  processes  of  their  program  may  be  exported/reused  directly  to  become 
cornerstones  of  SDI  metrics/quality  programs. 

The  submarine  systems  environment  has  several  similar  characteristics  to  SDI  component 
systems,  including  austere  survivability  and  reliability  requirements.  In  certain  cases,  the 
environment  and  operating  conditions  may  be  even  more  severe  since  maintenance  corrections  and 
fault  repairs  during  operational  conditions  at  sea  are  not  allowed.  In  many  instances,  reliability  and 
MTBF's  are  pushed  to  the  limit.  Large  megabit  transmission  links  are  non-  existant  to  effect 
downloading  of  new  or  enhanced  software  versions  as  may  be  found  in  SDI  space-based 
subsystems.  The  information  exchange  with  NUSC  is  in  its  initial  phase,  and  further  technical 
interchanges  are  envisioned. 

3.2  NAVAL  SURFACE  WARFARE  CENTER  -  DAHLCREN 

Dr.  William  Farr  at  NSWL  Strategic  Systems  Dept,  has  been  active  on  the  IEEE  Software 
Reliability  Measurement  Working  Group  Steering  Committee  and  a  contributor  to  the  IEEE  draft 
standard  P982.1  committee.  Dr.  Farr  has  developed  a  package  of  8  metrics  models  dubbed 
"SMERFS"  (Statistical  Modelling  and  Estimating  of  Reliability  Functions  for  Software)  and  last 
reported  over  50  different  users.  One  of  which,  NASA-Johnson  Space  Flight  Center,  will  be 
discussed  next.  (A  module  package  with  usage  documents  have  been  received— refer  to  Section 
5.5.2). 

3.3  IB.VI  AT  NASA-JOHNSON  SPACE  FLIGHT  CENTER 

David  Hamilton  of  IBM  at  NASA  in  Houston  has  been  a  SMERFS  user  and  reports  highly 
satisfactory  results  using  the  SMERFS-based  Schneidewind  Model.  He  has  had  one  years'  use 
with  the  SMERFS  package.  It  should  be  noted  that  SMERFS  is  a  post -facto  metrics  application. 

The  IBM  Space  Shuttle  Programs’  "On  Board  Primary  Software  Reliability  Prediction"  effort 
source  information  was  examined  for  insight  into  metrics  and  models  used.  Paraphrasing  their 
approach  -  "The  emphasis  of  this  program  centers  on  providing  good  software  via  the  removal  of 
failure  mode  causes  based  on  the  early  identification  of  design  flaws  at  the  'front-end'  of  the 
process  (i.e.,  requirements  &  design)  rather  than  'defense  (e.g.,  redundancy)'  against  them."  A 
key  IBM  approach  to  detecting  so^are  errors  is  to  use  Statistical  Modelling  of  detection  history 
data  in  a  configuration  management  database.  The  methodology  is  to  apply  the  SMERFS,  a  multi¬ 
model  interactive  computer  program,  to  appropriate  discrepancy  data  in  the  database.  Model 
validation  centers  on  predicting  reliability.  Much  of  the  data  used  for  analysis  is  derived  from 
previously  developed  software  that  is  repeatedly  analyzed,  catalogued  and  statistically  analyzed. 
The  SMERFS  models  are  then  used  to  find  corresponding  "form  and  fit  representations".  The 
model  that  best  fit  their  data  is  the  Schneidewind  Model  which  uses  exponentially  decreasing  error 
detection  rates  and  a  poisson  process  failure  detection. 

The  validity  of  this  approach,  apart  from  a  firont-end  emphasis,  depends  upon; 

a.  the  application  of  software  engineering  methodologies; 

b.  well-defined  processes; 


TBK 
•PAITTA 
:  AMI  1 
■RC 
ORA 

1  IMWTlMVDtf 

1  unaiimiTo 

TATUM 
■DC  ' 

THE  ANALYTIC  SCIENCES  CORPORATION 


c .  performing  labor  intensive  and  extensive  multi-path 
activities  (e.g.,  code  to  fiinction/vice-versa 
correlation,  backward  chaining  analysis) 

d.  "(a)  software  error  history  sufficiently  analyzed  to  allow  application  of  (the) 
reliability  model". 

Note;  In  the  domain  of  SDI  software,  c.  and  d.  are  not  practical  due  to  the  amount  and  complexity 
of  the  software,  and  the  lack  of  historical  data  available  relative  to  it.  Many  of  the  SDI  software 
applications  are  being  developed  for  the  first  time. 
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4.  CAPABILITIES,  LIMITATION  AND  DEFICIENCIES  OF  MODELS 


The  prioritization  of  SDS  software  metrics  requirements  appeared  to  be  high  for  Reliability, 
Survivability,  Integrity,  and  Efficiency  software  attributes,  "niere  was  moderate  or  medium 
priority  for  Maintainability,  Portability,  Usability,  and  Reusability,  in  that  order.  The  assessment 
of  the  software  metrics  maturity  presents  a  different  ordering:  Efficiency,  Reliability, 
Maintainability  begin  most  matured,  with  considerable  less  for  Survivability,  and  negligible,  if 
any,  support  for  Portability,  Usability  and  Reusability  attributes.  The  rest  of  this  section  will 
sununarize  the  capabilities  of  the  field  of  software  metrics  models  to  support  each  of  these  areas. 

4.1  GENERAL  LIMITATIONS 

[CONT86]  makes  some  general  caveats  after  introducing  the  notion  of  a  (composite)  "Quality 
of  So^are"  metric  (1.2.2).  The  book  gives  the  following  warnings: 

a.  Like  items  must  be  measured  and  compared  together  "apples  with  ^ples". 

b.  When  metrics  are  moved  from  one  environment  to  another,  they  should  be  re¬ 
calibrated. 

c .  Metrics  are  to  support  not  replace  management  savvy. 

d .  Software  metrics  are  poor  personnel  evaluators. 

e .  Any  metric  model  output  is  not  better  than  its  input. 

f .  The  metrics  program  cost  should  be  less  than  its  benefits. 

4.2  RELIABILITY  MODELS 

Both  the  Time  Between  Failures  (TBF)  and  the  Failure  Count  models  have  had  considerable 
evaluation  and  study.  Recently,  John  Musa  stated  tnat  failure  count  and  time  was  the  correct  basis 
for  reliability  assessment  and  prediction  as  opposed  to  fault  or  defect  counting  and  projection 
[MUSA8903].  However,  as  stated  in  [IDA  P-2132.  7.3]  "Current  software  reliability  technology 
suffers  from  some  fundamental  problems  and  limitations”.  This  paper  states  that  the  convenient 
adoption  of  hardware  reliability  measures  obscures  the  fundamental  difference  between  hardware 
and  software  reliability:  that  though  hardware  is  subject  to  physical  aging,  software  is  not.  Fresh 
analysis  into  the  nature  of  software  reliability  and  trustworthiness  by  Pamas  suggest  that  a 
combinatorial^robalistic  indicator  may  be  needed:  die  probability  that  no  critical  fault  remains. 

[IDA  P-2132]  also  raises  doubts  that  studies  based  upon  non-critical  fault  rates  in  software 
targets  of  average  reliability  are  deterministic  toward  predictions  of  criticality  in  highly  reliable 
so^are.  The  IDA  paper  goes  on  to  quote  Goel  at  the  IDA  Testing  and  Evaluation  Workshop  that 
available  approaches  are  too  simplistic,  and  they  cast  the  very  pessimistic  statement  "Because  of  the 
fundamental  limitations  of  current  software  reliability  assessment  methodologies  ...  further  work 
on  enhancing  current  methodolgies  (sic)  is  unlikely  to  yield  satisfactory  results."  (Assuming  that 
the  Pamas  criteria  is  used  --  determination  of  the  probability  of  a  failure-causing  fault.) 
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This  pessimistic  cast  of  the  IDA  paper  was  presaged  back  in  1986  by  Abdel-Ghaly,  Chan 
and  Littlewood,  in  their  review  of  ten  reliability  models.  The  authors  pointed  out  that  predictive 
quality  is  the  only  consideration  for  a  user  of  software  reliability  models  and  a  good  "model"  is  not 
sufficient  to  produce  good  predictions.  They  found  that  the  inference  mles  and  the  predictive 
procedure  were  significant  to  the  relative  success  of  the  various  models  [ABDE8609]. 

The  authors  present  an  approach  for  validating  candidate  models  for  a  specific  application. 
The  results  showed  that  there  was  no  single  "best  buy"  among  them.  The  ten  models  were; 
Jeltnski  -  Moranda,  Bayesian  Jelinski-Moranda,  Littlewood,  Bayesian  Littlewood,  Littlewood- 
Verral,  Keiller-Littlewood,  WeibuU  Order  Statistics,  Duane,  Goel-Okumoto  Model,  and  Littlewood 
NHPP. 

4.3  CO.MPLEXITY/MAINTAINABILITY 

More  recent  analysis  of  maintainability  predictors  using  measures  as  program  specification 
change  rates,  programmer's  skill  levels,  and  volume  of  design  documentation,  has  considerable 
effect  on  error  rates.  Both  [TAKA8901J  and  [GIBS8903}  showed  that  structural  differences  do 
impact  programmer's  performance.  Specifically,  system  improvements  result  in  considerable 
improvements  in  programmer's  performance.  Gibson  and  Senn's  study  investigated  this  by 
asking  three  experienced  professional  programmers  to  perform  three  maintenance  tasks  on  three 
functionally  equivalent,  but  different  in  complexity,  versions  of  a  COBOL  system. 

Six  metrics  were  used  in  this  study;  the  Halstead's  E,  McCabe’s  cyclomatic  complexity, 
Woodward's  K,  Gaffney's  Jumps,  Chen's  MIN  (Maximum  Intersects  Number),  and  Benyon- 
Tinker's  Cx.  The  metrics  reflected  both  the  improvements  in  the  system  (in  terms  of  ease  in 
maintenance  as  system  complexity  is  decreased)  as  well  as  in  programmer  maintenance 
performance.  The  Halstead's  E,  McCabe’s  Cyclomatic  Complexity,  Woodward's  K,  and 
Gaffney's  Jumps  reflect  a  decrease  in  complexity  with  improvement  in  system  structure  (control 
flow  complexity).  The  Chen's  MIN  and  Benyon-Tinker's  Cx  on  the  other  hand  reflect  the 
offsetting  increase  in  complexity  due  to  IF  nesting  and  number  of  modules. 

This  study  was  consistent  with  earlier  work  by  Evangelist  and  work  by  Basili,  Selby  and 
Philips,  investigating  the  validity  of  complexity  measures  as  independent  predictors  of  effort  and  or 
psychological  complexity  [EVAN84,  EVAN83.  BASI83J.  These  investigators  concluded  that  the 
metrics  by  themselves  were  no  better  than  lines-of-code,  but  when  the  personnel  being  studied  was 
held  constant  (single  programmer  studied  across  multiple  projects),  there  appeared  to  be  some 
relative  correlation  with  actual  work  and  times  to  complete. 

Since  the  metrics  have  shown  to  be  meaningful  in  studies  where  the  work  object  is  changed, 
but  the  programme'  performing  the  work  is  fixed,  further  work  studying  programmer  variables  is 
needed.  A  detailed  problem  solving  and  work  trait  profiling  system  may  be  prossible  to  account  for 
significant  programmer  differences,  and  be  a  basis  to  increase  the  validity  of  complexity  metrics  for 
maintainability  predictions. 
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4.4  EFFICIENCY  MODELS 

The  use  of  modeling  to  predict  sequential  software  efficiency  is  quite  mature.  Typically  a 
simulation  language  is  used  to  model  the  intended  software  achitecture.  As  development  proceeds, 
more  data  is  fed  into  the  simulator,  and  the  accuracy  of  the  predictions  improves.  General  purpose 
simulators  like  Simscript  and  GPSS  have  been  available  for  many  years,  as  have  texts  and  articles 
[ADKI7704],  [SCHN7704]. 

However,  as  pointed  out  by  the  IDA  team  in  their  review  of  testing  technology  available  for 
concurrent  and  real-time  programs,  the  methodology  is  relatively  new.  One  recent  technology  is 
the  TAGS  Simulation  Compiler  offered  by  Teledyne  Brown.  Another  is  Auto-G  from  Advanced 
Systems  Architectures  of  England.  A  third  is  Statemate  by  "iLOGI  X' ,  an  Israeli  vendor. 

However  promising  these  packages  are,  they  require  further  investigation  and  validation. 

4.5  EFFICIENCY/PRODLCTIVITY  MODELS 

Halstead's  original  "Software  Science  E"  was  a  model  to  forecast  programming  effort  based 
upon  complexity  and  size  work  parameters.  The  COCOMO  and  SECOMO  models  are  based  on 
lines  of  code  and  other  intermediate  cost  drivers.  These  models  have  been  in  use  on  many 
programs  and  should  be  considered  mature  {BOEH81  ],  however,  David  Card  and  others  have 
found  that  the  factors  of  change  in  the  software  maintenance  environment  may  not  be  so 
predictable. 

The  maintenance  cost  model  was  studied  in  ICARD8807J.  The  results  showed  that  the 
standard  maintenance  cost  models  based  on  lines  of  code  measures  proved  inadequate  in  estimating 
maintenance  cost.  This  is  due  to  the  different  productivity  rates  of  maintenance  and  the 
accumulation  effect  of  program  changes. 

4.6  INTEGRITY/SECURITY  MODELS 

The  existence  of  an  integrity  model  is  not  obvious,  but  both  the  Air  Force  and  Navy  have 
developed  systems  security  risk  assessment  methods  INEUG85,  pg.  4-74,  75],  which  may  be 
used  to  assess  system  integrity  factors  in  terms  of  annual  levels  of  loss.  Both  methods  are  fairly 
inaccurate,  using  "fuzzy-metrics"  (approximated  orders  of  magnitude).  This  concept  of  fuzzy 
metrics  may  have  some  application  to  the  assessment  of  integrity  as  may  other  elements  of  the 
security  models:  vulnerability  inventories,  threat  catalogues,  service  and  data  asset  valuation,  and 
time-based  exposure  projections.  Fairly  recent  research  into  fuzzy  systems  [NEG081J  and 
modeling  [NEG087]  theory  has  developed  the  mathematical  foundations  needed  to  refine  such 
applied  ri^  models,  but  much  more  study  is  needed. 

4.7  PORTABILITY,  USABILITY,  REUSABILITY 

Models  to  support  assessments  of  portability,  usability  and  reusability  are  not  available.  This 
realization  was  also  derived  by  the  SPARTA  team  [SPART8903].  These  attributes  were  ranked  as 
medium  in  the  aggregate  rarikings,  so  some  assessment  of  their  relative  value  for  research  and 
development  is  warranted. 
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4.8  COMPOSITE  METRICS 

Though  both  [CONT86J  and  [RADC878A]  present  a  grand  accumulation  formula  for 
assessing  the  many  quality  factors  and  arriving  at  a  signle  coefficient,  there  is  considerable 
skepticism  that  such  a  calculation  would  obscure  the  meaningfiilness  to  seasoned  managers. 
Another  method  is  outlined  in  [GOIC81]  and  [SING87]  —  Multicriteria  Modeling  and/or  Fuzzy 
Multicriteria  Modeling. 

[SING87,  pg.  1827]  presents  a  good  introductory  example.  For  a  software  quality  metrics 
application  refer  to  the  discussions  about  the  Naval  Underwater  Systems  Command. 
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5.  AVAILABLE  SOFTWARE  METRICS  SUPPORT 


This  section  presents  the  Subtask  3  survey,  features  assessment  and  recommendation  of 
available  software  metrics  tools  and  environments. 

5.1  EXISTING  TOOLS  AND  ENVIRONMENTS 

An  extensive  list  of  tools  and  environments  have  been  extracted  from  TBE's  TASQ  database 
for  examination.  A  more  extensive  list  of  tools  and  environments  reviewed  and  others  that  are 
under  review  is  contained  in  Appendix  C. 

From  this  task's  point  of  view,  two  categories  of  tools  have  been  established:  tools  direcdy 
supporting  a  metric  (e.g.,  McCabe,  Halstead,  Complexity)  and  ancillary  support  tools  (i.e.,  tools 
that  can  support  or  assist  in  the  collection  or  analysis  of  a  metric).  Ancillary  support  tools  consist 
of  the  following  categories: 

Program  Management  &  Support  Tools 

Computer-Aid^  Software  ^gineering  Tools 

Linkers,  Loaders,  Debuggers 

Test  Generators 

Spreadsheets 

Databases 

Environments 

Thus,  while  spreadsheets  such  as  LOTUS  1-2-3  can  be  extremely  useful  in  collecting  and 
classifying  metrics  and  measurements,  spreadsheets  are  not  considered  metrics  tools. 
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5.2  ANALYTIC  EVALUATION  OF  TOOLS/ENVIRONMENTS 
A  number  of  metrics  tools  have  been  identified: 


Tool  Source 


SMART 

USA-CECOM/TBE 

SMERFS 

NSWC-Dahlgren/Wm. 

McCabe  Metric 

Farr 

Intermetrics 

Halstead  Metric 

Intermetrics 

Complexity  Measures  Tool 

EVB 

Complexity  Metric 

Computer  Systems 

Analyze 

Design 

Autometric,  Inc. 

Complexity  Measure 

PD  SIMTEL  20 

ADADL 

Software  Systems 

Byron 

Design,  Inc. 
Intermetrics 

Analyze 

AdaSoft 

Adamat 

Dynamics  Research 

Complexity  Analysis  Tool 

Corp. 

McCabe  Associates 

Ada  Complexity  Analysis  Tool 
(ACAT),  (McCabe) 

Teledyne  Brown 

AdaPIC 

Engineering 

Arcadia  Consortium 

Logiscope 

Verilog 

COCOMO 

TRW-Redondo  Beach 

SECOMO 

RADC-DACS 

With  the  emphasis  for  the  Strategic  Defense  Initiatives  to  capitalize  on  modem  software 
engineering  metht^s  and  approaches,  and  the  Ada  Technologies  initiatives,  only  a  few  of  these 
tools  will  be  discussed  further. 
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5.2.1  Software  Management  and  Reporting  Tool  (SMART) 

The  SMART  package  is  targeted  for  deployment  at  the  U.S.  Army  Communications  - 
Electronics  Command  (CECOM)  July-August  1989.  The  first  program  it  will  be  applied  on  is  the 
Advanced  Field  Artillery  Tactical  Data  System  (AFATDS). 

The  software  metrics  goal  is  to  implement  the  Software  Management  and  Quality  Indicators 
as  per  AFSCP-800-43  and  AFSCP-8()0-14  in  an  efficient,  extensible  architecture.  The  software 
metrics  indicators  and  data  management  will  be  based  on  IBM  PC -compatible  machines  using  the 
popular  "dBase3"  formats  with  the  "Clipper"  data  base  language  system. 

Table  5-1  shows  some  of  the  data  tracked  by  SMART. 


TABLE  5-1 

OVERVIEW  OF  SMART  SOFTWARE  DATA 


TYPE  INDICATOR 

Deviations 

Waivers 

Software  Trouble  Reports 

Test  Incident  Forms 

Engineering  Change  Proposals 

Software  Improvement  Reports 

Software  Discrepancy  Reports 

Computer  Resource  Utilization 

Software  Development  Manpower 

Requirements  Definition  and  Stability 

Software  Progress  -  Development  and  Test 

Cost/Schedule  Deviations 

Software  Development  Tools 

Completeness 

Design  Stmcture 

Defect  Density 

Fault  Density 

Test  Coverage 

Software  Maturity 

Documentation 
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TABLE  5-2 

LIST  OF  SAMPLE  GRAPHICS  FOR  SMART 


COMPUTER  RESOURCE  UTILIZATIONS  (3) 

Primary  Memory 
Secondary  Memory 
CPU  Utilization 
I/O  Utilization 

SOFTWARE  DEVELOPMENT  MANPOWER  (WBS) 

REQUIREMENTS  DEFINITION  AND  STABILITY  (2  LEVELS) 
System;  CSCI 

DEVELOPMENT  AND  TEST 

Scheduled/Actual;  CSCI  &  CSC 

COST/SCHEDULE  DEVIATIONS 

SOFTWARE  DEVELOPMENT  TOOLS 

COMPLETENESS 

Requirements  %/month 
Implementation  %/month 

DESIGN  STRUCTURE 
Modularity 

Complexity/Dependency 

DEFECT  DENSITIES 
Requirements 
Design 
Code 

FAULT  (FAILURE)  DENSITY 

TEST  COVERAGE 

SOFTWARE  MATURITY 

DOCUMENT  TROUBLES  (BY  MODULE) 
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5.2.2  Statistical  Modeling  and  Estimation  of  Reliability 
Functions  for  Software  (SMERFS) 

SMERFS  was  developed  several  years  ago  as  an  aid  in  the  evaluation  of  software  reliability. 
In  its  original  design  it  was  targeted  for  mainframe  and  mini-computer  environments.  Since  then  it 
has  also  been  adapted  to  operate  on  micro-computers,  specifically  IBM-PC/XT  compatibles. 

The  current  version  of  SMERFS  has  incorporated  eight  software  reliability  models.  The 
models  include  the  following;  (1)  Musa's  Execution  Model,  (2)  Goel-Okumoto  Non- 
Homogeneous  Poisson  Model,  (3)  Adapted  Goel-Okumoto  Non-Homogeneous  Poisson  Model, 
(4)  Moranda's  Geometric  Model,  (5)  Schafer's  Generalized  Poisson  Model,  (6)  Schneidewind's 
Model,  (7)  Littlewood-Verrall  Bayesian  Model,  and  (8)  the  Brooks-Motley  Model. 

SMERFS  contains  a  driver  which  is  claimed  to  make  it  machine  independent.  The  driver  is  a 
subset  of  the  American  Standards  Institute  (ANSI)  specifications  for  the  FORTRAN  77  compiler. 
Several  user  selectable  options  are  available  within  the  driver  and  allow  the  system  to  be  configured 
to  produce:  better  predictions;  output  plots  and  catalogued  output  files.  Currently  SMERFS  is 
operational  on  three  main  computer  groups  at  the  Naval  Surface  Weapons  Center  (NSWC), 
Dahlgren,  VA.  The  three  computer  groups  include  the  CDC  CYBER  170/875,  the  Vaxcluster 
11/785,  and  a  large  number  of  IBM-compatible  PCs.  Dr.  William  H.  Farr,  of  NSWC,  and  Mr. 
Oliver  D.  Smith,  of  EG&G  Washington  Analytical  Services  Center,  Inc.  both  claim  that 
transferring  SMERFS  to  other  computers  should  be  very  easily  accomplished.  [FARR8812] 

Besides  containing  operating  instructions  within  its  interactive  mode,  two  additional  pieces  of 
documentation  are  available  for  use  with  SMERFS.  The  two  supplemental  reports  are:  (1) 
SMERFS  Library  Access  Guide  (NSWC-TR-84-37J ,  Rev.  1),  and  (2)  SMERFS  User's  Guide 
{NSWC-TR-84-373 ,  Rev.  1).  These  two  publications  aUow  a  potential  user  to  preview  the 
system.  Examples  are  provided  throughout  the  User's  Guide,  allowing  a  potential  user  to  acquire 
an  overview  of  the  SMERFS  processing.  In  addition,  the  guide  dso  shows  actual  software 
reliability  analyses  performed  on  the  CDC  CYBER  170/875. 

The  SMERFS  systems  show  tremendous  potential  for  use  in  the  SDI  environment  with  a 
minimum  of  modification. 

5.2.3  McCabe  Complexity  Tools 


Tools  based  on  McCabe  Complexity  Analysis  are  of  limited  use  in  the  Ada  domain,unless 
SDI  applications  use  older  implementation  languages  (e.g.,  COBOL,  CMS-2,  FORTRAN). 
Technical  Report  MC87-  McCabe  11-0003,  Extending  McCabe's  Cvclomatic  Complexity  Metric  for 
Analysis  of  Ada  Software.  [TBE8703]  U.S.  Army  AMCCOM;  Dover,  N.J.  under  contract 
DAAA21-85-D-0010  identifies  extensions  that  are  required  of  McCabe's  Theorems  if  Ada,  or  other 
modem  language  is  the  implementation  language.  McCabe  complexity  ^plications  are  limited  to 
pre-Ada  higher  order  languages  that  do  not  contain  modem  software  engineering  abstraction 
process  or  language  constructs  such  as  packages,  generics  or  tasking  mechanisms,  l^us,  pre-Ada 
tools  that  fall  into  this  category  (which  is  most  of  them)  will  require  updating. 


The  above  referenced  report  has  resulted  in  technical  report  MC87-McCabe  11-0005, 

''  '  the  Ada  Comolexitv  Analysis  Tool  which 
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AMCCOM;  Dover,  N.J.,  contract  DAAA21-85-D-  0010,  establishing  the  requirements  for  an  Ada 
Complexity  Analysis  Tool  (ACAT)  to  support  Ada  implementations. 


A  number  of  Ada  based  or  derived  program  design  language  (PDL)  analysis  tools  have 
emerged  over  the  past  several  years.  Two  of  those  listed,  Byron  and  ADADL,  are  mature  enough 
to  be  employed  to  perform  code  analysis  completeness  and  consistency  checking,  cross 
referencing,  structure  checking,  code  verification  completeness  and  correcmess,  and  interface 
analysis. 

ADADL  has  both  a  McCabe  and  an  ADADL  complexity  metric  for  both  pseudocode  and  the 
Ada  code  for  each  program  unit.  Both  of  these  complexity  metrics  have  not  been  evaluated  for 
degree  of  appropriateness. 

These  are  estimation  metrics  tool  used  on  pseudocode  or  Ada  (late  life  cycle  phase 
application).  It  should  also  be  noted  that  a  standard  DoD  Ada  PDL  does  not  and  may  never  exist, 
thus  PDL  tool  invocation  is  at  the  implementers  or  applications  level  discretion.  Automatic  code 
generation  or  a  shift  in  design  emphasis  to  an  SADMT  [IDA8804A&B]  or  like  representation  may 
obviate  the  need  for  such  in  the  future. 

5.2.4  Software  Engineering  Co.st  Model  (SECOMO) 

SECOMO  is  an  interactive  software  cost  estimation  tool,  based  on  the  COCOMO  cost  model, 
for  calculating  the  total  technical  and  support  manpower  requirements  of  a  Life  Cycle  Software 
Engineering  (LCSE)  Center.  SECOMO  is  maintained  and  distributed  by  the  Rome  Air 
Development  Centers  Data  and  Analysis  Center  for  Software  (DACS)  at  a  $50  charge. 

SECOMO  includes  a  "Care"  cost  limit  for  the  fire-up  phase  of  an  LCSE  Center. 

Developed  and  maintained  by  the  IIT  Research  Institute  and  RADC,  it  is  kept  current  with  the 
TRW/Barry  Boehm  COCOMO  users  days  recommendations. 

Significant  enhancements  are; 

•  An  Ada  parameters  set 

•  Pull  down  menus  and  user  efficiencies 

•  Site-timing  parameters 

Army  Materiel  Command  sites  which  are  user  include: 

•  CECOM 

•  AVSCOM 

•  AMCCOM 

•  MICOM 

Navy  sites  include:  NSWC-Dahlgren,  NUSC  -  New  London,  NCSC  -  Panama  City,  FL;  NTS  - 
Orlando. 
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5.2,5  Revised  Enhanced  Version  in  COCOMO  "RE VIC" 

REVIC  is  another  COCOMO  enhancement  maintained  by  an  Air  Force  Center.  The 
headquarters  command  at  Kirtland  AFB,  New  Mexico  has  developed  an  extensive  set  of  project 
performance  data  which  they  use  to  tune  the  REVIC  model.  Updates  and  enhancements  to  baseline 
COCOMO  and  to  SECOMO  are  brought  into  REVIC.  (No  Charge)  HQCMP/EPR,  Kirtland  AFB, 
NM  87117-5000. 

The  REVIC  User  Group  "RUG"  is  chaired  by  Dr.  George  Hozier  at  the  University  of  New 
Mexico. 

The  active  users  include  many  Air  Force  sites  and  contractors; 

Air  Force  Commands 

ESD,  RADC,  ACS-Pentagon 

Contractors 

The  Aerospace  Corporation 

Boeing 

Hughes 

GE 

TRW 

Lockheed 

Textron 


5.3  EXPERIENCE  ASSESSMENTS 

Adamat,  both  versions  of  Analyze,  and  Logiscope  are  mature  metrics  tools  for  extensive 
complexity  analysis.  Logiscope,  at  this  time,  does  not  suppon  Ada  source  code,  the  others  do. 
Adamat  appears  to  have  (based  on  literature  review)  the  most  extensive  set  of  metrics  criterion.  A 
partial  list  is  identified: 

Anomaly  Management:  Default  initialization,  basic  loops  containing  a  conditional 
exit  or  return,  strong  type  checking  and  constraint  checking,  raising  user  defined 
exceptions,  range  checking,  etc. 

Independence:  Isolation  of  input/output  routines,  isolation  of  tasking  statements 
and  declarations,  independence  from  system-  dependent  modules,  access  types, 
package  system,  use  of  length  clauses,  etc. 

Modularity:  Use  of  private  and  limited  private  types,  single  type  declarations  in 
package  specifications,  etc. 

Self-Descriptions :  Use  of  comments,  use  of  identifier  names,  etc. 

Simplicity:  Boolean  expressions,  labels,  decision  points,  branches,  branch 
constmcts,  nesting  levels,  use  of  literals,  procedure  calls,  etc. 
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System  Clarity:  Parenthesized  expressions,  no  default  mode  parameters,  access 
types  declared  private,  loops-modules-blocks  names,  etc. 

The  TASQ  and  ISEC  [TBE86091  tools/environment  databases  are  extensive  and  robust 
(containing  several  thousand  tools).  Initial  metrics  queries  have  resulted  in  a  sparse  tool 
identification.  A  formal  and  more  extensive  categorization  will  be  completed  by  mid-April,  and 
win  be  categorized  per  the  matrix  contained  in  Appendix  D  for  completeness.  All  tools  identified  in 
Appendix  C  will  also  be  matrix  categorized  as  such.  A  metrics  tools  expansion  list  much  beyond 
that  identified  in  this  section  is  not  expected.  Recent  dialogue  with  Ada  environment  developers 
and  tool  vendors  at  such  exhibits/expositions  as  TRIADA,  7th  Annual  National  Ada  Conference, 
CASE  EXPO,  NSIA,  etc.,  reveal  that  metrics  tools  are  sparse  and  not  the  focus  of  their 
development  efforts.  This  is  understandable  in  light  of  the  state  and  maturity  of  Ada  technology. 
The  latter  may  account  for  the  individual  DoD  service  initiatives  aimed  at  software  quality  programs 
and  tool  initiatives  such  as  those  identified  in  this  report  and  focusing  on  metrics.  The  SDI  will  in 
all  probability  require  its  own  tailored  metrics  and  quality  control  program  similar  to  those  of 
NUSC,  NASA  and  AMMCOM.  Much  of  the  present  developer/vendor  effort  is  directed  at 
environments,  compiler  maturation  and  efficiency,  and  related  support  tools  (e.g.,  linkers,  loaders, 
editors). 

Many  of  the  Ada  developers  at  this  time  are  looking  for  third  party  vendors  to  provide  them 
with  complementary/synergistic  metrics  tools  (Rational  and  Alsys  are  examples  of  such). 

The  emphasis  on  predictive  metrics  and  early  life  cycle  phase  emphasis  has  identified  major 
metrics  tool  deficiencies  in  these  areas  -  virtually  non-existent.  Well  defined  metrics,  measurand 
relationships  and  formalisms  in  the  early  phases  supported  by  tools  have  yet  to  appear  to  provide 
the  productivity  impact  they  are  expected  to  have.  The  AdaPIC  tool  set  (a  futures  project)  ongoing 
within  the  Arcadia  consortium,  holds  some  promise,  but  these  new  tools  have  yet  to  be  developed 
[WOLF8903].  Similarly,  TBE's  ACAT  is  under  development. 

Unfortunately  many  of  the  AdaPIC  tools  will  not  address  the  early  life  cycle  phases,  but  are 
aimed  at  complementing  the  emerging  development  environment.  Specifically,  more  systems 
engineering  metrics  tools  are  required  to  support  system  synthesis  and  couple  with  the  early 
software  engineering  processes. 

5.4  REQUISITE  ENVIRONMENT/TOOLSET  FEATURES 

As  stated  in  Section  5.3,  the  proposed  TASQ  format  (identified  in  the  matrix  of  Appendix  D) 
will  be  utilized  to  provide  the  next  elaboration  update  of  tools  contained  in  Appendix  C. 
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PROPOSED  IEEE  STANDARD  QUALITY  AND  RELIABILITY 
METRICS  TERMINOLOOY 


Term/Phrase 
Concept  Phase 

Critical  Range 
Critical  Value 
Defect 

Direct  Metric 

Error 


Factor  Sample 
Factor  Value 

Failure 


Definition 

The  period  of  time  in  the  software  life  cycle  during 
which  system  concepts  and  objectives  needed  by  the 
user  are  identified  and  documented.  Precedes  the 
Requirements  Phase. 

Metric  values  used  to  classify  software  into 
categories  of  acceptable,  marginal  and  unacceptable. 

Metric  value  of  a  validated  metric  which  is  used  to 
identify  software  which  has  unacceptable  quality. 

A  product  anomaly.  Examples  include  such  things 
as:  1)  omissions  and  imperfections  found  during 
early  life  cycle  phases  and  2)  faults  contained  in 
software  sufficiently  mature  for  test  or  operation. 
See  also  "fault". 

A  metric  that  represents  and  defines  a  software 
quality  factor,  and  which  is  valid  by  definition  (e.g., 
mean  time  to  software  failure  for  the  factor 
reliability). 

Human  action  that  results  in  software  containing  a 
fault.  Examples  include  omission  or 
misinterpretation  of  user  requirements  in  a  software 
specification,  incorrect  translation  or  omission  of  a 
requirement  in  the  design  specification  (ANSI/IEEE 
Std  729-1983). 

A  set  of  factor  values  which  is  drawn  from  the 
metrics  data  base  and  used  in  metrics  validation. 

A  value  (see  "metric  value")  of  the  direct  metric  that 
represents  a  factor. 

I )  The  termination  of  the  ability  of  a  functional  unit 
to  perform  its  required  function  (ISO;  ANSI/IEEE 
Std  729-1983).  2)  An  event  in  which  a  system  or 
system  component  does  not  perform  a  required 
function  within  specified  limits.  A  failure  may  be 
produced  when  a  fault  is  encountered  (ANSI/IEEE 
Std  729-1983). 


A-2 


Fault 


Measure 

Measurement 

Metrics  Framework 

Metrics  Sample 
Metric  Validation 

Metric  Value 

Predictive  Assessment 


1 )  An  accidental  condition  that  causes  a  functional 
unit  to  fail  to  perform  its  required  function  (ISO, 
ANSIAEEE  Std  729-1983).  2)  A  manifestation  of 
an  error  in  software.  A  fault,  if  encountered,  may 
cause  a  failure.  Synonymous  with  bug  (ANSIAEEE 
Std  729-1983). 

1)  A  quantitative  assessment  of  the  degree  to  which 
a  software  product  or  process  possesses  a  given 
attribute  (IEEE  P982.1/D3.0).  2)  To  ascertain  or 
appraise  by  comparing  to  a  standard;  to  apply  a 
nitric  (IEEE  PI 061 ). 

1)  The  act  or  process  of  measuring.  2)  A  figure, 
extent,  or  amount  obtained  by  measuring. 

A  tool  used  for  organizing,  selecting, 
communicating  and  evaluating  the  required  quality 
attributes  for  a  software  system;  a  hierarchical 
breakdown  of  factors,  subfactors  and  metrics  for  a 
software  system. 

A  set  of  metrics  values  which  is  drawn  from  the 
metrics  data  base  and  used  in  metrics  validation. 

The  act  or  process  of  ensuring  that  a  metric  correctly 
predias  or  accesses  a  quality  factor. 

An  element  from  the  range  of  a  metric;  a  metric 
output. 

The  process  of  using  a  predictor  metric(s)  to  predict 
the  value  of  another  metric. 


Predictive  Metric  A  metric  which  is  used  to  predict  the  values  of 

another  metric. 

Primitive  Data  relating  to  the  development  or  use  of  software 

that  is  used  in  developing  measures  or  quantitative 
descriptions  of  software.  Primitives  are  directly 
measurable  or  countable,  or  may  be  given  a  constant 
value  or  condition  for  a  specific  measure.  Examples 
include  error,  failure,  fault,  time,  time  interval,  date, 
number  of  noncommentary  source  code  statements, 
edges,  nodes. 

Any  task  performed  in  the  development, 
implementation  or  maintenance  of  software  (e.g., 
identify  the  software  components  of  a  system  as  part 
of  the  design). 


Process  Step 


Process  Metric 


Product  Metric 
Quality  Attribute 
Quality  Factor 
Quality  Requirement 

Quality  Sub-factor 
Sample  Software 

Software  Component 

Sofhi’are  Quality  Metric 

Software  Reliability 


Metrics  used  to  measure  characteristics  of  the 
methods,  techniques,  and  tools  employed  in 
acquiring,  developing,  verifying,  operating  and 
changing  the  software  system. 

Metrics  used  to  measure  the  characteristics  of  the 
documentation  and  code. 

A  characteristic  of  software;  a  generic  term  applying 
to  factors,  sub-factors,  or  metric  values. 

An  attribute  of  software  that  contributes  to  its 
quality. 

A  requirement  that  a  software  attribute  be  present  in 
software  to  satisfy  a  contract,  standard, 
specification,  or  other  formally  imposed. 

A  decomposition  of  a  quality  factor  or  quality  sub¬ 
factor  document. 

Software  selected  from  a  current  or  completed 
project  from  which  data  can  be  obtained  for  use  in 
preliminary  testing  of  data  collection  and  metric 
computation  procedures. 

General  term  used  to  refer  to  an  element  of  a 
software  system,  such  as  module,  unit,  data  or 
document. 

A  function  whose  inputs  are  software  data  and 
whose  output  is  a  single  (numerical)  value  that  can 
be  interpreted  as  the  degree  to  which  software 
possesses  a  given  attribute  that  affects  its  quality. 

The  probability  that  software  will  not  cause  the 
failure  of  a  system  for  a  specified  time  under 
specified  conditions.  The  probability  is  a  fimction 
of  the  inputs  to  and  use  of  the  system  as  well  as  a 
function  of  the  existence  of  faults  in  the  software. 
The  inputs  to  the  system  determined  whether 
existing  faults,  if  any,  are  encountered  (ANSI/IEEE 
Std  729-1983). 


Software  Reliability  Management 


The  process  of  optimizing  the  reliability  of  software 
through  a  program  which  emphasizes  software  error 
prevention,  fault  detection  and  removal,  and  the  use 
of  measurements  to  maximize  reliability  in  light  of 
project  constraints  such  as  resources  (cost), 
schedule,  and  performance. 

Validated  Metric  A  metric  whose  values  have  been  statistically 

associated  with  corresponding  quality  factor  values. 
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APPENDIX  B 

METRICS  RANKINCJ  TABLES  BY  SLBFLNCTION 


<  B-1  through  B-36  > 


Table  B-1  Subfunction  Ranked  Attributes  i  Metric  Models 


Subfunction  Title:  Detect  Plumes 


SUBFUNCTION  ATTRIBUTES, 

RANKINGS  t, 

SCORES 

Reli 

Surv 

Intg 

Thru 

Usab 

Main 

Port 

Reus 

Effc 

H 

H 

H 

H 

L 

L 

L 

L 

M 

Time  Between  Failure 

5 

5 

5 

5 

1 

1 

1 

1 

3 

Jelinski/Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton  (Linear) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton  (Parabolic) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Geometric  De-eut) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Hybrid  Geomet  Poiss) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Littlewood/Ver rail 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Lloyd/Lipow 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Complexity 

Halstead 

5 

0 

0 

0 

0 

1 

0 

0 

0 

McCabe 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Woodward  (Knot  Counts) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Chen  (Nested  Decision  Stmts) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Gaffney 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Benyc  i-Tinker 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Gilb's  (Binary  Decision) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Chapin's  Q 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Segment-Global  Usage  Pair 
Myer's  (McCabe  extension) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Hansen's  (McCabe/Halstead) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Oviedo's  (Data/Ctrl  Flows) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Failure  Count 

Geol/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Schneidewind 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Geol 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Musa 

5 

0. 

0 

0 

0 

0 

0 

0 

0 

Shooman 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Jelinski/Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto  (Gen  Poisson) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Brooks/Motley 

5 

5 

0 

0 

0 

0 

0 

0 

0 

IBM  Poisson 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Fault  Seeding 

Mills 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Input  Domain  Based 

Nelson 

5 

0 

0 

0 

0 

0 

0 

0 

■M 

Ho 

5 

0 

0 

0 

0 

0 

0 

Bl 

Ramamoo  r  thy/Bastani 

5 

■1 

0 

0 

0 

0 

0 

0 

Table  B-2  Subfunction  Ranked  Attributes  &  Metric  Models 


Subfunction  Title:  Detect  Cold  Bodies 


SUBFUNCTION  ATTRIBUTES, 

RANKINGS  i 

SCORES 

Mt'^rDTr*  r»T  hccr'c  r  uAnn  c 

Reli 

Surv 

Intg 

Thru 

Usab 

Main 

Port 

Reus 

E££c 

H 

H 

H 

H 

L 

L 

L 

L 

H 

Time  Between  Failure 

5 

5 

5 

5 

1 

1 

1 

1 

5 

Jel inski /Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton  (Linear) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton  (Parabolic) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Geometric  De-eut) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Hybrid  Geomet  Poiss) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Littlewood/Ver rail 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Lloyd/Lipow 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Complexity 

Halstead 

5 

0 

0 

0 

0 

1 

0 

0 

0 

McCabe 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Woodward  (Knot  Counts) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Chen  (Nested  Decision  Stmts) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Gaffney 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Benyon-Tinker 

Gilb's  (Binary  Decision) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Chapin's  Q 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Segment-Global  Usage  Pair 
Myer's  (McCabe  extension) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Hansen's  (McCabe/Halstead) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Oviedo's  (Data/Ctrl  Flows) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Failure  Count 

Geol/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Schneidewind 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Geol 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Musa 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Shooman 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Jelinski/Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto  (Gen  Poisson) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Brooks/Motley 

5 

5 

0 

0 

0 

0 

0 

0 

0 

IBM  Poisson 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Fault  Seeding 

Mills 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Input  Domain  Based 

Nelson 

5 

0 

0 

0 

0 

0 

0 

0 

Ho 

5 

0 

0 

0 

0 

0 

Ramamoorthy/Bastani 

5 

0 

0 

0 

0 

0 

0 

■I 

Table  B-3  Subfunction  Ranked  Attributes  &  Metric  Models 
Subfunction  Title:  RF  Detect 


SUBFUNCTION  ATTRIBUTES, 

RANKINGS  i 

SCORES 

Reli 

Surv 

Intg 

Thru 

Usab 

Main 

Port 

Reus 

Effc 

K 

H 

H 

H 

L 

M 

M 

L 

M 

Time  Between  Failure 

5 

5 

5 

5 

1 

3 

3 

1 

3 

Jel inski /Mora nda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton  (Linear) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton  (Parabolic) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Geometric  De-eut) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Hybrid  Geomet  Poiss) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Littlewood/Ver rail 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Lloyd/Lipow 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Complexity 

Halstead 

5 

0 

0 

0 

0 

3 

0 

0 

0 

McCabe 

5 

0 

0 

0 

0 

3 

0 

0 

0 

Woodward  (Knot  Counts) 

5 

0 

0 

0 

0 

3 

0 

0 

0 

Chen  (Nested  Decision  Stmts) 

5 

0 

0 

0 

0 

3 

0 

0 

0 

Gaffney 

5 

0 

0 

0 

0 

3 

0 

0 

0 

Benyon-Tinker 

5 

0 

0 

0 

0 

3 

0 

0 

0 

Gilb's  (Binary  Decision) 

5 

0 

0 

0 

0 

3 

0 

0 

0 

Chapin's  Q 

5 

0 

0 

0 

0 

3 

0 

0 

0 

Segment-Global  Usage  Pair 
Myer's  (McCabe  extension) 

5 

0 

0 

0 

0 

3 

0 

0 

0 

5 

0 

0 

0 

0 

3 

0 

0 

0 

Hansen's  (McCabe/Halstead) 

5 

0 

0 

0 

0 

3 

0 

0 

0 

Oviedo's  (Data/Ctrl  Flows) 

5 

0 

0 

0 

0 

3 

0 

0 

0 

Failure  Count 

Geol/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Schneidewind 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Geol 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Musa 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Shooman 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Jel inski /Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto  (Gen  Poisson) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Brooks/Motley 

5 

5 

0 

0 

0 

0 

0 

0 

0 

IBM  Poisson 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Fault  Seeding 

Mills 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Input  Domain  Based 

00000000 

00000000 

00000000 


Nelson 

Ho 

Ramamoorthy/Bastani 


5 

5 

5 


Table  B-4  Subfunction  Ranked  Attributes  &  Metric  Models 


Subfunction  Title:  Resolve  Objects 


SUBFUNCTION  ATTRIBUTES,  RANKINGS  6  SCORES 

METRIC  CLASSES  &  MODELS  - 

-  Reli  Surv  Intg  Thru  Usab  Main  Port  Reus  Effc 


HHHMLLLLM 


Time  Between  Failure 

5 

5 

5 

3 

1 

1 

1 

1 

3 

Jel inski /Mo randa 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton  (Linear) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton  (Parabolic) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Geometric  De-eut) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Hybrid  Geomet  Poiss) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Littlewood/Ver rail 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Lloyd/Lipow 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Complexity 

Halstead 

5 

0 

0 

0 

0 

1 

0 

0 

0 

McCabe 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Woodward  (Knot  Counts) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Chen  (Nested  Decision  Stmts) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Gaffney 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Benyon-Tinker 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Gilb's  (Binary  Decision) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Chapin’s  Q 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Segment-Global  Usage  Pair 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Myer's  (McCabe  extension) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Hansen's  (McCabe/Halstead) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Oviedo's  (Data/Ctrl  Flows) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Failure  Count 

Geol/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Schneidewind 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Geol 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Musa 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Shooman 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Jel inski /Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto  (Gen  Poisson) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Brooks/Motley 

5 

5 

0 

0 

0 

0 

0 

0 

0 

IBM  Poisson 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Fault  Seeding 

Mills 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Input  Domain  Based 

Nelson 

5 

0 

0 

0 

0 

0 

0 

0 

Ho 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Ramamoorthy/Bastani 

5 

0 

0 

0 

0 

0 

0 

0 

Table  B-5  Subfunction  Ranked  Attributes  &  Metric  Models 


Subfunction  Title:  Discriminate 


SUBFUNCTION  ATTRIBUTES, 

RANKINGS  & 

SCORES 

MC'T'DTP  r*T  hCCCC  r  UAniT*!  C 

* 

Reli 

Surv 

Intg 

Thru 

Usab 

Main 

Port 

Reus 

Effc 

H 

H 

H 

M 

L 

L 

L 

L 

M 

Time  Between  Failure 

5 

5 

5 

3 

1 

1 

1 

1 

3 

Jelinski/Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton  (Linear) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton  (Parabolic) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Geometric  De-eut) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Hybrid  Geomet  Poiss) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

L i 1 1 1 ewood/Ve  r  r  a 1 1 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Lloyd/Lipow 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Complexity 

Halstead 

5 

0 

0 

0 

0 

1 

0 

0 

0 

McCabe 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Woodward  (Knot  Counts) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Chen  (Nested  Decision  Stmts) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Gaffney 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Benyon-Tinker 

Gilb's  (Binary  Decision) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Chapin's  Q 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Segment-Global  Usage  Pair 
Myer's  (McCabe  extension) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Hansen's  (McCabe/Halstead) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Oviedo's  (Data/Ctrl  Flows) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Failure  Count 

Geol/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Schneidewind 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Geol 

5 

0 

.  0 

0 

0 

0 

0 

0 

0 

Musa 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Shooman 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Jelinski/Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto  (Gen  Poisson) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Brooks/Motley 

5 

5 

0 

0 

0 

0 

0 

0 

0 

IBM  Poisson 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Fault  Seeding 

Mills 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Input  Domain  Based 

Nelson 

5 

0 

0 

0 

0 

0 

0 

0 

■M 

Ho 

5 

0 

0 

0 

0 

0 

0 

0 

Ramamoorthy/Bastani 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Table  B-6  Subfunction  Ranked  Attributes  &  Metric  Models 


Subfunction  Title:  Assess  Kills 


SUBFUNCTION  ATTRIBUTES, 

RANKINGS  f, 

SCORES 

** 

Reli 

Surv 

Intg 

Thru 

Usab 

Main 

Port 

Reus 

Effc 

H 

H 

H 

M 

M 

M 

M 

L 

M 

Time  Between  Failure 

5 

5 

5 

3 

3 

3 

3 

1 

3 

Jelinski/Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton  (Linear) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton  (Parabolic) 

6 

5 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Geometric  De-eut) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Hybrid  Geomet  Poiss) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Littlewood/Ver rail 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Lloyd/Lipow 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Complexity 

Hal::tead 

5 

0 

0 

0 

0 

3 

0 

0 

0 

McCaoe 

5 

0 

0 

0 

0 

3 

0 

0 

0 

Woodward  (Knot  Counts) 

5 

0 

0 

0 

0 

3 

0 

0 

0 

Chen  (Nested  Decision  Stmts) 

5 

0 

0 

0 

0 

3 

0 

0 

0 

Gaffney 

5 

0 

0 

0 

0 

3 

0 

0 

0 

Benyon-Tinker 

5 

0 

0 

0 

0 

3 

0 

0 

0 

Gilb's  (Binary  Decision) 

5 

0 

0 

0 

0 

3 

0 

0 

0 

Chapin's  Q 

5 

0 

0 

0 

0 

3 

0 

0 

0 

Segment-Global  Usage  Pair 

5 

0 

0 

0 

0 

3 

0 

0 

0 

Myer's  (McCabe  extension) 

5 

0 

0 

0 

0 

3 

0 

0 

0 

Hansen's  (McCabe/Halstead) 

5 

0 

0 

0 

0 

3 

0 

0 

0 

Oviedo's  (Data/Ctrl  Flows) 

5 

0 

0 

0 

0 

3 

0 

0 

0 

Failure  Count 

Geol/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Schneidewind 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Geol 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Musa 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Shooman 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Jelinski/Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto  (Gen  Poisson) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Brooks/Motley 

5 

5 

0 

0 

0 

0 

0 

0 

0 

IBM  Poisson 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Fault  Seeding 

Mills 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Input  Domain  Based 

Nelson 

5 

0 

wm 

0 

0 

0 

0 

0 

0 

Ho 

5 

0 

Bl 

0 

0 

0 

0 

0 

0 

Ramamoor thy/Bastani 

5 

0 

0 

0 

0 

0 

0 

0 

0 

1 

^  Table  B-7  Subfunction  Ranked 

Attributes 

i,  Metric  Models 

1 

■  Subfunction  Title:  Correlate 

1 

SUBFUNCT I ON  ATTR I BUTES , 

RANKINGS  & 

SCORES  1 

"  METRIC  CLASSES  &  MODELS 

— 

— 

— 

- - 

— 

m 

Reli 

Surv 

Intg 

Thru 

Usab 

Main 

Port 

Reus 

Effc 

1 

H 

H 

H 

H 

L 

L 

L 

L 

M 

Time  Between  Failure 

5 

5 

5 

5 

1 

1 

1 

1 

3 

■  Jel inski/Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

■  Schick/Wolver ton  (Linear) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton  (Parabolic) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

H  Moranda  (Geometric  De-eut) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

1  Moranda  (Hybrid  Geomet  Poiss) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

*  Goel/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Littlewood/Ver rail 

5 

0 

0 

0 

0 

0 

0 

0 

0 

■  Lloyd/Lipow 

5 

0 

0 

0 

0 

0 

0 

0 

0 

■ 

Complexity 

I  Halstead 

5 

0 

0 

0 

0 

1 

0 

0 

0 

■  McCabe 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Woodward  (Knot  Counts) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

A  Chen  (Nested  Decision  Stmts) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

■  Gaffney 

5 

0 

0 

0 

0 

1 

0 

0 

0 

1  *  Benyon-Tinker 

5 

0 

0 

0 

0 

1 

0 

0 

0 

1  Gilb's  (Binary  Decision) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

1  ■  Chapin's  Q 

5 

0 

0 

0 

0 

1 

0 

0 

0 

1  1  Segment-Global  Usage  Pair 

5 

0 

0 

0 

0 

1 

0 

0 

0 

1  Myer’s  iMcCabe  extension) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

I  ^  Hansen's  (McCabe/Halstead) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

I  1  Oviedo's  (Data/Ctrl  Flows) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

■  Failure  Count 

I  ■  Geol/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

1  *  Schneidewind 

5 

0 

0 

0 

0 

0 

0 

0 

0 

1  Geol 

5 

0 

0 

0 

0 

0 

0 

0 

0 

■  |l  Musa 

5 

0. 

0 

0 

0 

0 

0 

0 

0 

1  1  Shooman 

5 

0 

0 

0 

0 

0 

0 

0 

0 

1  Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

■  _  Jelinski/Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

■  1  Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

■  "  Schick/Wolverton 

5 

5 

0 

0 

0 

0 

0 

0 

0 

■  Goel/Okumoto  (Gen  Poisson) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

I  n  Brooks/Motley 

5 

5 

0 

0 

0 

0 

0 

0 

0 

1  I  IBM  Poisson 

5 

5 

0 

0 

0 

0 

0 

0 

0 

■  Fault  Seeding 

1  1  Mills 

5 

0 

0 

0 

0 

0 

0 

0 

0 

1  _  Input  Domain  Based 

I 

I 


Nelson 

Ho 

Ramamoorthy/Bastani 


5 

5 

5 


Table  B-8  Subfunction  Ranked  Attributes  &  Metric  Models 


Subfunction  Title:  Initiate  Track 


SUBFUNCTION  ATTRIBUTES, 

RANKINGS  & 

SCORES 

METRIC  CLASSES  &  MODELS 

Reli 

Surv 

Intg 

Thru 

Usab 

Main 

Port 

Reus 

Effc 

H 

H 

H 

H 

L 

L 

L 

L 

M 

Time  Between  Failure 

5 

5 

5 

5 

1 

1 

1 

1 

3 

Jelinski/Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton  (Linear) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton  (Parabolic) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Geometric  De-eut) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Hybrid  Geomet  Poiss) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Littlewood/Verrall 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Lloyd/Lipow 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Complexi ty 

Halstead 

5 

0 

0 

0 

0 

1 

0 

0 

0 

McCabe 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Woodward  (Knot  Counts) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Chen  (Nested  Decision  Stmts) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Gaffney 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Benyon-Tinker 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Gilb's  (Binary  Decision) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Chapin's  Q 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Segment-Global  Usage  Pair 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Myer's  (McCabe  extension) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Hansen's  (McCabe/Halstead) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Oviedo's  (Data/Ctrl  Flows) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Failure  Count 

Geol/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Schneidewind 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Geol 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Musa 

5 

c 

0 

0 

0 

0 

0 

0 

0 

Shooman 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Jelinski/Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto  (Gen  Poisson) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Brooks/Motley 

5 

5 

0 

0 

0 

0 

0 

0 

0 

IBM  Poisson 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Fault  Seeding 

Mills 

5 

0 

0 

0 

0 

0 

0 

0 

Input  Domain  Based 

Nelson 

5 

C 

0 

wm 

0 

0 

0 

0 

0 

Ho 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Ramamoor thy/Bastani 

5 

0 

0 

0 

0 

0 

0 

0 

1 

^  Table  B-9  Subfunction  Ranked 

Attributes 

&  Metric  Models 

■  Subfunction  Title:  Est 

imate  State 

1 

SUBFUNCTION  ATTRIBUTES, 

RANKINGS  & 

SCORES 

^  METRIC  CLASSES  &  MODELS 

— 

— 

— 

— 

— 

— 

- - - 

Reli 

Surv 

Intg 

Thru 

Usab 

Main 

Port 

Reus 

Effc 

1 

H 

H 

H 

H 

L 

L 

M 

L 

M 

Time  Between  Failure 

5 

5 

5 

5 

1 

1 

3 

1 

3 

1  Jelinski/Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

P  Schick/Wolver ton  (Linear) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton  (Parabolic) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Geometric  De-eut) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

■  Moranda  (Hybrid  Geomet  Poiss 

)  5 

0 

0 

0 

0 

0 

0 

0 

0 

*  Goel/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Littlewood/Ver  rail 

5 

0 

0 

0 

0 

0 

0 

0 

0 

■  Lloyd/Lipow 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Complex i ty 

1  Halstead 

5 

0 

0 

0 

0 

1 

0 

0 

0 

■  McCabe 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Woodward  (Knot  Counts) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Chen  (Nested  Decision  Stmts) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

■  Gaffney 

5 

0 

0 

0 

0 

1 

0 

0 

0 

"  Benyon-Tinker 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Gilb's  (Binary  Decision) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

fl  Chapin's  Q 

5 

0 

0 

0 

0 

1 

0 

0 

0 

H  Segment-Global  Usage  Pair 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Myer's  (McCabe  extension) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Hansen's  (McCabe/Halstead) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

■  Oviedo's  (Data/Ctrl  Flows) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Failure  Count 

■  Geol/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

"  Schneidewind 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Geol 

5 

0 

0 

0 

0 

0 

0 

0 

0 

■  Musa 

5 

Q 

0 

0 

0 

0 

0 

0 

0 

■  Shooman 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

.  Jelinski/Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

■  Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

■  Schick/Wolver ton 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto  (Gen  Poisson) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

■  Brooks/Motley 

5 

5 

0 

0 

0 

0 

0 

0 

0 

1  IBM  Poisson 

5 

5 

0 

0 

0 

0 

0 

0 

0 

^  Fault  Seeding 

1  Mills 

5 

0 

0 

0 

0 

0 

0 

0 

0 

^  Input  Domain  Based 

I 

I 


Nelson 

Ho 

Ramamoor thy/Bastani 


5 

5 

5 


Table  B-IO  Subfunction  Ranked  Attributes  &  Metric  Models 


I 


Subfunction  Title:  Predict  Intercept  and  Impact  Points 


I 

I 


METRIC  CLASSES  &  MODELS 


SUBFUNCTION  ATTRIBUTES,  RANKINGS  6  SCORES 
Reli  Surv  Intg  Thru  Usab  Main  Port  Reus  E££c 


H 


H 


H 


M 


Time  Between  Failure 


Jelinski/Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton  (Linear) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton  (Parabolic) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Geometric  De-eut) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Hybrid  Geomet  Poiss) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Li 1 1 1 ewood/Ve  r  rail 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Lloyd/Lipow 

5 

0 

0 

0 

0 

0 

0 

0 

O' 

Complexity 

Halstead 

5 

0 

0 

0 

0 

1 

0 

0 

0 

McCabe 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Woodward  (Knot  Counts) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Chen  (Nested  Decision  Stmts) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Gaffney 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Benyon-Tinker 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Gilb's  (Binary  Decision) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Chapin's  Q 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Segment-Global  Usage  Pair 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Myer's  (McCabe  extension) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Hansen's  (McCabe/Halstead) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Oviedo's  (Data/Ctrl  Flows) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Failure  Count 

Geol/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Schneidewind 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Geol 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Musa 

5 

0  . 

0 

0 

0 

0 

0 

0 

0 

Shooman 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Jelinski/Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto  (Gen  Poisson) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Brooks/Motley 

5 

5 

0 

0 

0 

0 

0 

0 

0 

IBM  Poisson 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Fault  Seeding 

Mills 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Input  Domain  Based 

Nelson 

5 

0 

Mi 

0 

0 

n 

0 

0 

0 

Ho 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Ramamoorthy/Bastani 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Table  B-11  Subfunction  Ranked  Attributes  k  Metric  Models 


Subfunction  Title:  Interplatform  Data  Communications 


SUBFUNCTION  ATTRIBUTES, 

RANKINGS  & 

SCORES 

ut'T'OTr'  nr  nccrc  c  urvnrr  c 

i  X  X  ^  0i 

Reli 

Surv 

Intg 

Thru 

Usab 

Main 

Port 

Reus 

Effc 

H 

H 

H 

H 

L 

L 

L 

M 

H 

Time  Between  Failure 

5 

5 

5 

5 

1 

1 

1 

3 

5 

Jel inski /Mo randa 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wol ver ton  (Linear) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton  (Parabolic) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Geometric  De-eut) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Hybrid  Geomet  Poiss) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Littlewood/Ver rail 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Lloyd/Lipow 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Complexity 

Halstead 

5 

0 

0 

0 

0 

1 

0 

0 

0 

McCabe 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Woodward  (Knot  Counts) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Chen  (Nested  Decision  Stmts) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Gaffney 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Benyon-Tinker 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Gilb's  (Binary  Decision) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Chapin's  Q 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Segment-Global  Usage  Pair 
Myer's  (McCabe  extension) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Hansen's  (McCabe/Halstead) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Oviedo's  (Data/Ctrl  Flows) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Failure  Count 

Geol/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Schneidewind 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Geol 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Musa 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Shooman 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Jel inski /Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wol ver ton 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto  (Gen  Poisson) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Brooks/Motley 

5 

5 

0 

0 

0 

0 

0 

0 

0 

IBM  Poisson 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Fault  Seeding 

Mills 

5 

0 

0 

0 

w 

0 

0 

0 

0 

Input  Domain  Based 

Table  B-12  Subfunction  Ranked  Attributes  &  Metric  Models 


Subfunction  Title:  Ground  -  Space  Communications 


SUBFUNCTION  ATTRIBUTES, 

RANKINGS  fc 

SCORES 

METRIC  CLASSES  &  MODELS 

. 

Reli 

Surv 

Intg 

Thru 

Usab 

Main 

Port 

Reus 

Effc 

H 

H 

H 

M 

L 

M 

M 

L 

M 

Time  Between  Failure 

5 

5 

5 

3 

1 

3 

3 

1 

3 

Je 1 inski /Mo randa 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton  (Linear) 

5 

5 

0 

0 

u 

0 

0 

0 

0 

Schick/Wolver ton  (Parabolic) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Geometric  De-eut) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Hybrid  Geomet  Poiss) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Li t  tlewood/Ver rail 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Llcyd/Lipow 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Complexity 

Halstead 

5 

0 

0 

0 

0 

3 

0 

0 

0 

McCabe 

5 

0 

0 

0 

0 

3 

0 

0 

0 

Woodward  (Knot  Counts) 

6 

0 

0 

0 

0 

3 

0 

0 

0 

Chen  (Nested  Decision  Stmts) 

5 

0 

0 

0 

0 

3 

0 

0 

0 

Gaffney 

5 

0 

0 

0 

0 

3 

0 

0 

0 

Benyon-Tinker 

5 

0 

0 

0 

0 

3 

0 

0 

0 

Gilb's  (Binary  Decision) 
Chapin's  Q 

5 

0 

0 

0 

0 

3 

0 

0 

0 

5 

0 

0 

0 

0 

3 

0 

0 

0 

Segment-Global  Usage  Pair 
Kyer's  (McCabe  extension) 

5 

0 

0 

0 

0 

3 

0 

0 

0 

5 

0 

0 

0 

0 

3 

0 

0 

0 

Hansen's  (McCabe/Halstead) 

5 

0 

0 

0 

0 

3 

0 

0 

0 

Oviedo's  (Data/Ctrl  Flows) 

5 

0 

0 

0 

0 

3 

0 

0 

0 

Failure  Count 

Geol/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Schneidewind 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Geol 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Musa 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Shooman 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Jel inski /Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto  (Gen  Poisson) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Brooks/Motley 

5 

5 

0 

0 

0 

0 

0 

0 

0 

IBM  Poisson 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Fault  Seeding 

Mills 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Input  Domain  Based 

Kelson 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Ho 

5 

0 

0 

0 

0 

0 

0 

0 

Ramamoorthy/Bastani 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Table  B-13  Subfunction  Ranked  Attributes  &  Metric  Models 


Subfunction  Title:  Ground  Communications 


SUBFUNCTION  ATTRIBUTES,  RANKINGS  &  SCORES 

METRIC  CLASSES  &  MODELS  - 

-  Reli  Surv  intg  Thru  Usab  Main  Port  Reus  Effc 


MMMMMHMHL 


Time  Between  Failure 

3 

3 

3 

3 

3 

5 

3 

5 

1 

Jelinski/Moranda 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton  (Linear) 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton  (Parabolic) 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Geometric  De-eut) 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Hybrid  Geomet  Poiss) 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Littlewood/Ver rail 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Lloyd/Lipow 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Complexity 

Halstead 

3 

0 

0 

0 

0 

5 

0 

0 

0 

McCabe 

3 

0 

0 

0 

0 

5 

0 

0 

0 

Woodward  (Knot  Counts) 

3 

0 

0 

0 

0 

5 

0 

0 

0 

Chen  (Nested  Decision  Stmts) 

3 

0 

0 

0 

0 

5 

0 

0 

0 

Gaffney 

3 

0 

0 

0 

0 

5 

0 

0 

0 

Benyon-Tinker 

3 

0 

0 

0 

0 

5 

0 

0 

0 

Gilb's  (Binary  Decision) 

3 

0 

0 

0 

0 

5 

0 

0 

0 

Chapin's  Q 

3 

0 

0 

0 

0 

5 

0 

0 

0 

Segment-Global  Usage  Pair 
Myer's  (McCabe  extension) 

3 

0 

0 

0 

0 

5 

0 

0 

0 

3 

0 

0 

0 

0 

5 

0 

0 

0 

Hansen's  (McCabe/Halstead) 

3 

0 

0 

0 

0 

5 

0 

0 

0 

Oviedo's  (Data/Ctrl  Flows) 

3 

0 

0 

0 

0 

5 

0 

0 

0 

Failure  Count 

Geol/Okumoto 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Schneidewind 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Geol 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Musa 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Shooman 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Jelinski/Moranda 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Moranda 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto  (Gen  Poisson] 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Brooks/Motley 

3 

3 

0 

0 

0 

0 

0 

0 

0 

IBM  Poisson 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Fault  Seeding 

fills 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Input  Domain  Based 

Nelson 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Ho 

3 

0 

0 

0 

0 

0 

0 

0 

mm 

Ramamoorthy/Bastani 

3 

0 

0 

0 

0 

0 

0 

0 

■1 

1 

Table  B-16  Subfunction  Ranked 

Attributes 

&  Metric 

Models 

1 

Subfunction  Title:  Assign  and 

Control 

5BI  Weapons 

1 

SUBFUNCTION  ATTRIBUTES,  RANKINGS  & 

SCORES  1 

■ 

METRIC  CLASSES  &  MODELS 

— 

Reli 

Surv 

Intg 

Thru 

Usab  Main 

Port 

Reus 

Effc 

1 

H 

H 

H 

H 

L 

L 

L 

L 

M 

■ 

Time  Between  Failure 

5 

5 

5 

5 

1 

1 

1 

1 

3 

m 

Jelinski/Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

1 

Schick/Wolver ton  (Linear) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton  (Parabolic) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Geometric  De-eut) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

■ 

Moranda  (Hybrid  Geomet  Poiss) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

1 

Goel/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Littlewood/Ver rail 

5 

0 

0 

0 

0 

0 

0 

0 

0 

■ 

Lloyd/Lipow 

5 

0 

0 

0 

0 

0 

0 

0 

0 

1 

Complexity 

1 

Halstead 

5 

0 

0 

0 

0 

1 

0 

0 

0 

1  I 

McCabe 

5 

0 

0 

0 

0 

1 

0 

0 

0 

1  * 

Woodward  (Knot  Counts) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Chen  (Nested  Decision  Stmts) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

1  n 

Gaffney 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Benyon-Tinker 

5 

0 

0 

0 

0 

1 

0 

0 

0 

1 

Gilb's  (Binary  Decision) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

1  ^ 

Chapin's  Q 

5 

0 

0 

0 

0 

1 

0 

0 

0 

1  I 

Segment-Global  Usage  Pair 

5 

0 

0 

0 

0 

1 

0 

0 

0 

1  B 

Myer's  (McCabe  extension) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

1 

Hansen's  (McCabe/Halstead) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

1  ■ 

Oviedo's  (Data/Ctrl  Flows) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

1  * 

Failure  Count 

1  m 

Geol/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Schneidewind 

5 

0 

0 

0 

0 

0 

0 

0 

0 

1 

Geol 

5 

0 

0 

0 

0 

0 

0 

0 

0 

1 

Musa 

5 

0 

0 

0 

0 

0 

0 

0 

0 

H  ■ 

Shooman 

5 

0 

0 

0 

0 

0 

0 

0 

0 

■  B 

Moranda 

5 

c 

0 

0 

0 

0 

0 

0 

0 

B 

Jelinski/Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

B  fl| 

Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

B  B 

Schick/Wolverton 

5 

5 

0 

0 

0 

0 

0 

0 

0 

B 

Goel/Okumoto  (Gen  Poisson) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

B 

Brooks/Motley 

5 

5 

0 

0 

0 

0 

0 

0 

0 

B  fl 

IBM  Poisson 

5 

5 

0 

0 

0 

0 

0 

0 

0 

1 

Fault  Seeding 

1  1 

Mills 

5 

0 

0 

0 

0 

0 

0 

0 

0 

1 

Input  Domain  Based 

I  A 

Nelson 

5 

0 

0 

0 

0 

0 

0 

0 

0 

1  * 

Ho 

5 

0 

0 

0 

0 

0 

0 

0 

0 

|! 

Ramamoor thy/Bastani 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Table  B-17  Subfunction  Ranked  Attributes  &  Metric  Models 


Subfunction  Title:  Assign  and  Control  GBI  Weapons 


SUBFUNCTION  ATTRIBUTES, 

RANKINGS  S, 

SCORES 

METRIC  CLASSES  &  MODELS 

Reli 

Surv 

Intg 

Thru 

Usab 

Main 

Port 

Reus 

Effc 

H 

H 

H 

M 

L 

H 

L 

L 

M 

Time  Between  Failure 

5 

5 

5 

3 

1 

5 

1 

1 

3 

Jelinski/Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton  (Linear) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton  (Parabolic) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Geometric  De-eut) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Hybrid  Geomet  Poiss) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Littlewood/Ver rail 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Lloyd/Lipow 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Complexity 

Halstead 

5 

0 

0 

0 

0 

5 

0 

0 

0 

McCabe 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Woodward  (Knot  Counts) 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Chen  (Nested  Decision  Stmts) 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Gaffney 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Benyon-Tinker 

Gilb's  (Binary  Decision) 

5 

0 

0 

0 

0 

5 

0 

0 

0 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Chapin's  Q 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Segment-Global  Usage  Pair 
Myer’s  (McCabe  extension) 

5 

0 

0 

0 

0 

5 

0 

0 

0 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Hansen's  (McCabe/Halstead) 

5 

•  0 

0 

0 

0 

5 

0 

0 

0 

Oviedo's  (Data/Ctrl  Flows) 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Failure  Count 

Geol/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Schneidewind 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Geol 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Musa 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Shooman 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Jelinski/Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto  (Gen  Poisson) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Brooks/Motley 

5 

5 

0 

0 

0 

0 

0 

0 

0 

IBM  Poisson 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Fault  Seeding 

Mills 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Input  Domain  Based 

Nelson 

5 

0 

0 

0 

0 

0 

0 

0 

Ho 

5 

0 

0 

0 

0 

0 

0 

0 

Ramamoor thy/Bastani 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Table  B-18  Subfunction  Ranked  Attributes  &  Metric  Models 


Subfunction  Title:  Guide  and  Control  SBI  Weapons 


SUBFUNCTION  ATTRIBUTES, 

RANKINGS  & 

SCORES 

liirmoT/^  m  h.cc'C'C  t  Lmnrr  c 

P  M  W  B 

Reli 

Surv 

Intg 

Thru 

Usab 

Main 

Port 

Reus 

Effc 

H 

H 

H 

H 

L 

L 

L 

L 

H 

Time  Between  Failure 

5 

5 

5 

5 

1 

1 

1 

1 

5 

Jel inski /Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton  (Linear) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton  (Parabolic) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Geometric  De-eut) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Hybrid  Geomet  Poiss) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Littlewood/Verrall 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Lloyd/Lipow 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Complexity 

Halstead 

5 

0 

0 

0 

0 

1 

0 

0 

0 

McCabe 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Woodward  (Knot  Counts) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Chen  (Nested  Decision  Stmts) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Gaffney 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Benyon-Tinker 

Gilb's  (Binary  Decision) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Chapin's  Q 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Segment-Global  Usage  Pair 
Myer's  (McCabe  extension) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Hansen's  (McCabe/Halstead) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Oviedo's  (Data/Ctrl  Flows) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Failure  Count 

Geol/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Schneidewind 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Geol 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Musa 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Shooman 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Jelinski/Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto  (Gen  Poisson) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Brooks/Motley 

5 

5 

0 

0 

0 

0 

0 

0 

0 

IBM  Poisson 

5 

5 

0 

0 

0 

0 

0 

V 

0 

Fault  Seeding 

Mills 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Input  Domain  Based 

Nelson 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Ho 

5 

0 

0 

0 

0 

0 

0 

0 

Ramamoorthy/Bastani 

5 

0 

0 

0 

0 

0 

0 

Table  B-19  Subfunction  Ranked  Attributes  &  Metric  Models 


Subfunction  Title:  Guide  and  Control  GBI  Weapons 


SUBFUNCTION  ATTRIBUTES,  RANKINGS  &  SCORES 

METRIC  CLASSES  &  MODELS  - 

-  Reli  Surv  Intg  Thru  Usab  Main  Port  Reus  Effc 


HHHMLHLLM 


Time  Between  Failure 

5 

5 

5 

3 

1 

5 

1 

1 

3 

Jelinski/Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton  (Linear) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton  (Parabolic) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Geometric  De-eut) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Hybrid  Geomet  Poiss) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Littlewood/Ver rail 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Lloyd/Lipow 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Complexity 

Halstead 

5 

0 

0 

0 

0 

5 

0 

0 

0 

McCabe 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Woodward  (Knot  Counts) 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Chen  (Nested  Decision  Stmts) 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Gaffney 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Benyon-Tinker 

Gilb's  (Binary  Decision) 

5 

0 

0 

0 

0 

5 

0 

0 

0 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Chapin's  Q 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Segment-Global  Usage  Pair 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Myer's  (McCabe  extension) 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Hansen's  (McCabe/Halstead) 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Oviedo's  {Data/Ctrl  Flows) 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Failure  Count 

Geol/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Schneidewind 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Geol 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Musa 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Shooman 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Jelinski/Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto  (Gen  Poisson) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Brooks/Motley 

5 

5 

0 

0 

0 

0 

0 

0 

0 

IBM  Poisson 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Fault  Seeding 

Mills 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Input  Domain  Based 

Nelson 

5 

0 

0 

n 

0 

0 

0 

0 

0 

Ho 

5 

0 

0 

0 

0 

0 

0 

0 

Ramamoorthy/Bastani 

5 

0 

0 

0 

0 

0 

0 

0 

Table  B-20  Subfunction  Ranked  Attributes  &  Metric  Models 
Subfunction  Title:  Command  Environment  Control 


SUBFUNCTION  ATTRIBUTES, 

RANKINGS  f, 

SCORES 

Reli 

Surv 

Intg 

Thru 

Usab 

Main 

Port 

Reus 

E£c 

H 

M 

H 

M 

M 

H 

M 

L 

M 

Time  Between  Failure 

5 

3 

5 

3 

3 

5 

3 

1 

3 

Jelinski/Moranda 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton  (Linear) 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton  (Parabolic) 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Geometric  De-eut) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Hybrid  Geomet  Poiss) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Li ttlewood/Verrall 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Lloyd/Lipow 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Complexity 

Halstead 

5 

0 

0 

0 

0 

5 

0 

0 

0 

McCabe 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Woodward  (Knot  Counts) 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Chen  (Nested  Decision  Stmts) 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Gaffney 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Benyon-Tinker 

Gilb's  (Binary  Decision) 

5 

0 

0 

0 

0 

5 

0 

0 

0 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Chapin's  Q 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Segment-Global  Usage  Pair 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Myer's  (McCabe  extension) 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Hansen's  (McCabe/Halstead) 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Oviedo's  (Data/Ctrl  Flows) 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Failure  Count 

Geol/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Schneidewind 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Geol 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Musa 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Shooman 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

3 

0 

0 

0 

0 

0 

0 

-Jelinski/Moranda 

5 

3 

0 

0 

0 

n 

0 

0 

0 

Moranda 

5 

3 

0 

0 

0 

c 

0 

0 

0 

Schick/Wolverton 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Goel/Okurooto  (Gen  Poisson) 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Brooks/Motley 

5 

3 

0 

0 

0 

0 

0 

0 

0 

IBM  Poisson 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Fault  Seeding 

Mills 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Input  Domain  Based 

Nelson 

5 

0 

■n 

0 

0 

0 

0 

0 

Ho 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Ramamoorthy/Bastani 

5 

0 

0 

0 

0 

0 

0 

I 


Table  B-21  Subfunction  Ranked  Attributes  &  Metric  Models 


Subfunction  Title:  Control  Onboard  Environment 


SUBPUNCTION  ATTRIBUTES,  RANKINGS  6  SCORES 

METRIC  CLASSES  &  MODELS  - 

-  Reli  Surv  Intg  Thru  Usab  Main  Port  Reus  Effc 


HHHHLLLLM 
Time  Between  Failure  555511113 


Jelinski/Moranda 
Schick/Wolver ton  (Linear) 
Schick/Wolverton  (Parabolic) 
Moranda  (Geometric  De-eut) 
Moranda  (Hybrid  Geomet  Poiss) 
Goel/Okumoto 
Littlewood/Ver rail 
Lloyd/Lipow 

Complexity 


5  5  0 
5  5  0 
5  5  0 
5  0  0 
5  0  0 
5  0  0 
5  0  0 
5  0  0 


0  0  0 
0  0  0 
0  0  0 
0  0  0 
0  0  0 
0  0  0 
0  0  0 
0  0  0 


0  0  0 
0  0  0 
0  0  0 
0  0  0 
0  0  0 
0  0  0 
0  0  0 
0  0  0 


Halstead 

McCabe 

Woodward  (Knot  Counts) 

Chen  (Nested  Decision  Stmts) 

Gaffney 

Benyon-Tinker 

Gilb's  (Binary  Decision) 

Chapin's  0 

Segment-Global  Usage  Pair 
Myer's  (McCabe  extension) 
Hansen's  (McCabe/Halstead) 
Oviedo's  (Data/Ctrl  Flows) 

Failure  Count 


5  0  0 
5  0  0 
5  0  0 
5  0  0 
5  0  0 
5  0  0 
5  0  0 
5  0  0 
5  0  0 
5  0  0 
5  0  0 
5  0  0 


0  0  1 
0  0  1 
0  0  1 
0  0  1 
0  0  1 
0  0  1 
0  0  1 
0  0  1 
0  0  1 
0  0  1 
0  0  1 
0  0  1 


0  0  0 
0  0  0 
0  0  0 
0  0  0 
0  0  0 
0  0  0 
0  0  0 
0  0  0 
0  0  0 
0  0  0 
0  0  0 
0  0  0 


Geol/Okumoto 

Schneidewind 

Geol 

Musa 

Shooman 

Moranda 

Jelinski/Moranda 

Moranda 

Schick/Wolverton 
Goel/Okumoto  (Gen  Poisson) 
Brooks/Motley 
IBM  Poisson 

Fault  Seeding 


5  0  0 
5  0  0 
5  0  0 
5  0  0 
5  0  0 
5  5  0 
5  5  0 
5  5  0 
5  5  0 
5  5  0 
5  5  0 
5  5  0 


0  0  0 
0  0  0 
0  0  0 
0  0  0 
0  0  0 
0  0  0 
0  0  0 
0  0  0 
0  0  0 
0  0  0 
0  0  0 
0  0  0 


0  0  0 

0  0  c 

0  0  0 

0  0  0 

0  0  0 

0  0  0 

0  0  0 

0  0  0 

0  0  0 

0  0  0 

0  0  0 

0  0  0 


Mills  500000000 

Input  Domain  Based 


Nelson  500000000 

Ho  500000000 

Ramamoorthy/Bastani  500000000 


Table  B-22  Subfunction  Ranked  Attributes  (  Metric  Models 


Subfunction  Title:  Command  Attitude  and  Position  Control 


SUBFUNCTION  ATTRIBUTES,  RANKINGS  S,  SCORES 

METRIC  CLASSES  &  MODELS  - 

-  Reli  Surv  Intg  Thru  Usab  Main  Port  Reus  Effc 


HMHMMHMLM 


Time  Between  Failure 

5 

3 

5 

3 

3 

5 

3 

1 

3 

Jel inski /Moranda 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton  (Linear) 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton  (Parabolic) 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Geometric  De-eut) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Hybrid  Geomet  Poiss) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Littlewood/Ver rail 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Lloyd/Lipow 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Complexity 

Halstead 

5 

0 

0 

0 

0 

5 

0 

0 

0 

McCabe 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Woodward  (Knot  Counts) 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Chen  (Nested  Decision  Stmts) 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Gaffney 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Benyon-Tinker 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Gilb's  (Binary  Decision) 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Chapin's  Q 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Segment-Global  Usage  Pair 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Myer's  (McCabe  extension) 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Hansen's  (McCabe/Halstead) 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Oviedo's  (Data/Ctrl  Flows) 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Failure  Count 

Geol/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Schneidewind 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Geol 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Husa 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Shooman 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Jel inski /Moranda 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto  (Gen  Poisson) 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Brooks/Motley 

5 

3 

0 

0 

0 

0 

0 

0 

0 

IBM  Poisson 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Fault  Seeding 

Mills 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Input  Domain  Based 

Nelson 

5 

0 

0 

0 

0 

0 

Bi 

m 

Ho 

5 

0 

0 

0 

0 

0 

WM 

0 

Ramamoorthy/Bastani 

5 

0 

0 

0 

0 

0 

0 

0 

Table  B-23  Subfunction  Ranked  Attributes  &  Metric  Models 


Subfunction  Title:  Control  Onboard  Attitude  and  Position 


SUBFUNCTION  ATTRIBUTES,  RANKINGS  6  SCORES 

METRIC  CLASSES  &  MODELS  - 

-  Reli  Surv  Intg  Thru  Usab  Main  Port  Reus  Effc 


HHHHLLLLM 


Time  Between  Failure 

5 

5 

5 

5 

1 

1 

1 

1 

3 

Jelinski/Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton  (Linear) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton  (Parabolic) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Geometric  De-eut) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Hybrid  Geomet  Poiss) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Littlewood/Ver rail 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Lloyd/Lipow 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Complexity 

Halstead 

5 

0 

0 

0 

0 

1 

0 

0 

0 

McCabe 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Woodward  (Knot  Counts) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Chen  (Nested  Decision  Stmts) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Gaffney 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Benyon-Tinker 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Gilb's  (Binary  Decision) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Chapin's  0 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Segment-Global  Usage  Pair 

Myer's  (McCabe  extension) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Hansen's  (McCabe/Halstead) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Oviedo's  (Data/Ctrl  Flows) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Failure  Count 

Geol/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Schneidewind 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Geol 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Musa 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Shooman 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Jelinski/Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto  (Gen  Poisson) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Brooks/Motley 

5 

5 

0 

0 

0 

0 

0 

0 

0 

IBM  Poisson 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Fault  Seeding 

Mills 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Input  Domain  Based 

Nelson 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Ho 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Ramamoorthy/Bastani 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Table  B-24  Subfunction  Ranked  Attributes  fc  Metric  Models 


Subfunction  Title:  Sense  Onboard  Status 


SUBFUNCTION  ATTRIBUTES, 

RANKINGS  t 

SCORES 

Accc'c  r  unniTT  c 

WMnbwLaM  Qr  *  « 5 

Reli 

Surv 

Intg 

Thru 

Usab 

Main 

Port 

Reus 

Effc 

H 

H 

H 

M 

L 

L 

L 

L 

M 

Time  Between  Failure 

5 

5 

5 

3 

1 

1 

1 

1 

3 

Jelinski/Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton  (Linear) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton  (Parabolic) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Geometric  De-eut) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Hybrid  Geomet  Poiss) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Littlewood/Verrall 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Lloyd/Lipow 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Complexity 

Halstead 

5 

0 

0 

0 

0 

1 

0 

0 

0 

McCabe 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Woodward  (Knot  Counts) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Chen  (Nested  Decision  Stmts) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Gaffney 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Benyon-Tinker 

Gilb's  (Binary  Decision) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Chapin's  0 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Segment-Global  Usage  Pair 
Myer's  (McCabe  extension) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Hansen's  (McCabe/Halstead) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Oviedo's  (Data/Ctrl  Flows) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Failure  Count 

Geol/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Schneidewind 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Geol 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Musa 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Shooman 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Jelinski/Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto  (Gen  Poisson) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Brooks/Motley 

5 

5 

0 

0 

0 

0 

0 

0 

0 

IBM  Poisson 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Fault  Seeding 

Mills 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Input  Domain  Based 

Nelson 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Ho 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Ramamoorthy/Bastani 

5 

0 

0 

0 

0 

0 

0 

0 

Table  B-25  Subfunction  Ranked  Attributes  fc  Metric  Models 


Subfunction  Title:  Assess  Status 


SUBPUNCTION  ATTRIBUTES,  RANKINGS  t  SCORES 

METRIC  CLASSES  &  MODELS  - 

-  Reli  Surv  Intg  Thru  Usab  Main  Port  Reus  Effc 


HMHMMHMLM 


Time  Between  Failure 

5 

3 

5 

3 

3 

5 

3 

1 

3 

Jelinski/Moranda 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton  (Linear) 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton  (Parabolic) 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Geometric  De-eut) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Hybrid  Geomet  Poiss) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Littlewood/Verrall 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Lloyd/Lipow 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Complexity 

Halstead 

5 

0 

0 

0 

0 

5 

0 

0 

0 

McCabe 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Woodward  (Knot  Counts) 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Chen  (Nested  Decision  Stmts) 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Gaffney 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Benyon-Tinker 

Gilb's  (Binary  Decision) 

5 

0 

0 

0 

0 

5 

0 

0 

0 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Chapin's  Q 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Segment-Global  Usage  Pair 
Myer's  (McCabe  extension) 

5 

0 

0 

0 

0 

5 

0 

0 

0 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Hansen's  (McCabe/Halstead) 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Oviedo's  (Data/Ctrl  Flows) 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Failure  Count 

Geol/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Schneidewind 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Geol 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Musa 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Shooman 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Jelinski/Moranda 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto  (Gen  Poisson) 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Brooks/Motley 

5 

3 

0 

0 

0 

0 

0 

0 

0 

IBM  Poisson 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Fault  Seeding 

Mills 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Input  Domain  Based 

Nelson 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Ho 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Ramamoorthy/Bastani 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Table  B-26  Subfunction  Ranked  Attributes  i  Metric  Models 
Subfunction  Title:  Command  Reconfiguration 


SUBFUNCTION  ATTRIBUTES,  RANKINGS  &  SCORES 

METRIC  CLASSES  &  MODELS  - 

-  Reli  Surv  Intg  Thru  Usab  Main  Port  Reus  Effc 


KMHMMHMLM 


Time  Between  Failure 

5 

3 

5 

3 

3 

5 

3 

1 

3 

Jelinski/Moranda 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton  (Linear) 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton  (Parabolic) 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Geometric  De-eut) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Hybrid  Geomet  Poiss) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Littlewood/Verrall 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Lloyd/Lipow 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Complexity 

Halstead 

5 

0 

0 

0 

0 

5 

0 

0 

0 

McCabe 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Woodward  (Knot  Counts) 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Chen  (Nested  Decision  Stmts) 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Gaffney 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Benyon-Tinker 

Gilb's  (Binary  Decision) 

5 

0 

0 

0 

0 

5 

0 

0 

0 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Chapin's  Q 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Segment-Global  Usage  Pair 
Myer's  (McCabe  extension) 

5 

0 

0 

0 

0 

5 

0 

0 

0 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Hansen's  (McCabe/Halstead) 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Oviedo's  (Data/Ctrl  Flows) 

5 

0 

0 

0 

0 

5 

0 

0 

0 

Failure  Count 

Geol/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Schneidewind 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Geol 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Musa 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Shooman 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Jelinski/Moranda 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto  (Gen  Poisson) 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Brooks/Motley 

5 

3 

0 

0 

0 

0 

0 

0 

0 

IBM  Poisson 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Fault  Seeding 

Mills 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Input  Domain  Based 

Nelson 

5 

0 

0 

0 

0 

0 

0 

mm 

0 

Ho 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Ramamoor thy/Bastani 

5 

0 

0 

0 

0 

0 

0 

0 

Table  B-27  Subfunction  Ranked  Attributes  &  Metric  Models 
Subfunction  Title:  Reconfiguration 


SUBFUNCTION  ATTRIBUTES, 

RANKINGS  & 

SCORES 

un'OTr*  nr  Acet'c  r  unnirr  c 

Reli 

Surv 

Intg 

Thru 

Usab 

Main 

Port 

Reus 

Effc 

H 

H 

H 

H 

L 

L 

L 

L 

M 

Time  Between  Failure 

5 

S 

5 

5 

1 

1 

1 

1 

3 

Je 1 inski /Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton  (Linear) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton  (Parabolic) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Geometric  De-eut) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Hybrid  Geomet  Poiss) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Littlewood/Ver rail 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Lloyd/Lipow 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Complexity 

Halstead 

5 

0 

0 

0 

0 

1 

0 

0 

0 

McCabe 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Woodward  (Knot  Counts) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Chen  (Nested  Decision  Stmts) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Gaffney 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Benyon-Tinker 

Gilb's  (Binary  Decision) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Chapin’s  0 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Segment-Global  Usage  Pair 
Myer's  (McCabe  extension) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Hansen's  (McCabe/Halstead) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Oviedo's  {Data/Ctrl  Flows) 

5 

0 

0 

0 

0 

1 

0 

0 

0 

Failure  Count 

Geol/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Schneidewind 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Geol 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Musa 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Shooman 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Jelinski/Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto  (Gen  Poisson) 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Brooks/Motley 

5 

5 

0 

0 

0 

0 

0 

0 

0 

IBM  Poisson 

5 

5 

0 

0 

0 

0 

0 

0 

0 

Fault  Seeding 

Mills 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Input  Domain  Based 

Nelson 

5 

0 

0 

0 

MB 

0 

0 

0 

0 

Ho 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Ramamoorthy/Bastani 

5 

0 

0 

0 

0 

0 

0 

Table  B-28  Subfunction  Ranked  Attributes  &  Metric  Models 
Subfunction  Title:  Development  Tools 


SUBFUNCTION  ATTRIBUTES, 

RANKINGS  & 

SCORES 

Mr'TOTr*  rr  Accrue  r  unnin  c 

Reli 

Surv 

Intg 

Thru 

Usab 

Main 

Port 

Reus 

Effc 

M 

M 

M 

L 

M 

M 

M 

M 

M 

Time  Between  Failure 

3 

3 

3 

1 

3 

3 

3 

3 

3 

Jelinski/Moranda 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton  (Linear) 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton  (Parabolic) 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Geometric  De*eut) 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Hybrid  Geomet  Poiss) 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Li ttlewood/Ver rail 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Lloyd/Lipow 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Complexity 

Halstead 

3 

0 

0 

0 

0 

3 

0 

0 

0 

McCabe 

3 

0 

0 

0 

0 

3 

0 

0 

0 

Woodward  (Knot  Counts) 

3 

0 

0 

0 

0 

3 

0 

0 

0 

Chen  (Nested  Decision  Stmts) 

3 

0 

0 

0 

0 

3 

0 

0 

0 

Gaffney 

3 

0 

0 

0 

0 

3 

0 

0 

0 

Benyon-Tinker 

Gilb's  (Binary  Decision) 

3 

0 

0 

0 

0 

3 

0 

0 

0 

3 

0 

0 

0 

0 

3 

0 

0 

0 

Chapin's  Q 

3 

0 

0 

0 

0 

3 

0 

0 

0 

Segment-Global  Usage  Pair 
Myer's  (McCabe  extension) 

3 

0 

0 

0 

0 

3 

0 

0 

0 

3 

0 

0 

0 

0 

3 

0 

0 

0 

Hansen's  (McCabe/Halstead) 

3 

0 

0 

0 

0 

3 

0 

0 

0 

Oviedo's  (Data/Ctrl  Flows) 

3 

0 

0 

0 

0 

3 

0 

0 

0 

Failure  Count 

Geol/Okumoto 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Schneidewind 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Geol 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Musa 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Shooman 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Jelinski/Moranda 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Moranda 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto  (Gen  Poisson) 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Brooks/Motley 

3 

3 

0 

0 

0 

0 

0 

0 

0 

IBM  Poisson 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Fault  Seeding 

Mills 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Input  Domain  Based 

Nelson 

3 

■0 

0 

■D 

■Q 

0 

0 

0 

0 

Ho 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Ramamoorthy/Bastani 

3 

0 

0 

0 

0 

0 

0 

1 

—  Table  B-29  Subfunction  Ranked 

Attributes 

f.  Metric 

Models 

■  Subfunction  Title:  Hardware- 

Ln-the 

-loop 

(HWIL)  Simulation 

1 

SUBFUNCT I ON  ATTR I BUTES , 

RANKINGS  & 

SCORES  fl 

"  METRIC  CLASSES  &  MODELS 

— 

— 

— 

— 

- - — 

Reli 

Surv 

Intg  Thru 

Usab 

Main 

Port 

Reus 

Effc 

M 

H 

H 

M 

M 

M 

L 

L 

M 

Time  Between  Failure 

3 

5 

5 

3 

3 

3 

1 

1 

3 

■  Jelinski/Moranda 

3 

5 

0 

0 

0 

0 

0 

0 

0 

■  Schick/Wolverton  (Linear) 

3 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton  (Parabolic) 

3 

5 

0 

0 

0 

0 

0 

0 

0 

M  Moranda  (Geometric  De-eut) 

3 

0 

0 

0 

0 

0 

0 

0 

0 

■  Moranda  (Hybrid  Geomet  Poiss) 

3 

0 

0 

0 

0 

0 

0 

0 

0 

*  Goel/Okumoto 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Li ttlewood/Verrall 

3 

0 

0 

0 

0 

0 

0 

0 

c 

■  Lloyd/Lipow 

3 

0 

0 

0 

0 

0 

0 

0 

0 

B 

Complexity 

B  Halstead 

3 

0 

0 

0 

0 

3 

0 

0 

0 

B  McCabe 

3 

0 

0 

0 

0 

3 

0 

0 

0 

Woodward  (Knot  Counts) 

3 

0 

0 

0 

0 

3 

0 

0 

0 

M  Chen  (Nested  Decision  Stmts) 

3 

0 

0 

0 

0 

3 

0 

0 

0 

B  Gaffney 

3 

0 

0 

0 

0 

3 

0 

0 

0 

*  Benyon-Tinker 

3 

0 

0 

0 

0 

3 

0 

0 

0 

Gilb's  (Binary  Decision) 

3 

0 

0 

0 

0 

3 

0 

0 

0 

B  Chapin's  Q 

3 

0 

0 

0 

0 

3 

0 

0 

0 

B  Segment-Global  Usage  Pair 

3 

0 

0 

0 

0 

3 

0 

0 

0 

Myer's  (McCabe  extension) 

3 

0 

0 

0 

0 

3 

0 

0 

0 

^  Hansen's  (McCabe/Halstead) 

3 

0 

0 

0 

0 

3 

0 

0 

0 

fl  Oviedo's  (Data/Ctrl  Flows) 

3 

0 

0 

0 

0 

3 

0 

0 

Failure  Count 

. . 

Geol/Okumoto 

Schneidewind 

Geol 

Musa 

Shooman 

Moranda 

Jelinski/Moranda 

Moranda 

Schick/Wolverton 
Goel/Okumoto  (Gen  Poisson) 
Brooks/Motley 
IBM  Poisson 

Fault  Seeding 


Table  B-30  Subfunction  Ranked  Attributes  &  Metric  Models 


Subfunction  Title:  Demonstration  Simulation 


SUBFUNCTION  ATTRIBUTES, 

RANKINGS  & 

SCORES 

Reli 

Surv 

Intg  Thru 

Usab 

Main 

Port 

Reus 

Effc 

M 

H 

M 

M 

M 

M 

M 

L 

M 

Time  Between  Failure 

3 

5 

3 

3 

3 

3 

3 

1 

3 

Jelinski/Moranda 

3 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton  (Linear) 

3 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton  (Parabolic) 

3 

5 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Geometric  De-eut) 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Hybrid  Geomet  Poiss) 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Littlewood/Ver rail 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Lloyd/Lipow 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Complexity 

Halstead 

3 

0 

0 

0 

0 

3 

0 

0 

0 

McCabe 

3 

0 

0 

0 

0 

3 

0 

0 

0 

Woodward  (Knot  Counts) 

3 

0 

0 

0 

0 

3 

0 

0 

0 

Chen  (Nested  Decision  Stmts) 

3 

0 

0 

0 

0 

3 

0 

0 

0 

Gaffney 

3 

0 

0 

0 

0 

3 

0 

0 

0 

Benyon-Tinker 

Gilb's  (Binary  Decision) 

3 

0 

0 

0 

0 

3 

0 

0 

0 

3 

0 

0 

0 

0 

3 

0 

0 

0 

Chapin's  Q 

3 

0 

0 

0 

0 

3 

0 

0 

0 

Segment-Global  Usage  Pair 
Myer's  (McCabe  extension) 

3 

0 

0 

0 

0 

3 

0 

0 

0 

3 

0 

0 

0 

0 

3 

0 

0 

0 

Hansen's  (McCabe/Halstead) 

3 

0 

0 

0 

0 

3 

0 

0 

0 

Oviedo's  (Data/Ctrl  Flows) 

3 

0 

0 

0 

0 

3 

0 

0 

0 

Failure  Count 

Geol/Okumoto 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Schneidewind 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Geol 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Musa 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Shooman 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda 

3 

5 

0 

0 

0 

0 

0 

0 

0 

Jelinski/Moranda 

3 

5 

0 

0 

0 

0 

0 

0 

0 

Moranda 

3 

5 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton 

3 

5 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto  (Gen  Poisson) 

3 

5 

0 

0 

0 

0 

0 

0 

0 

Brooks/Motley 

3 

5 

0 

0 

0 

0 

0 

0 

0 

IBM  Poisson 

3 

5 

0 

0 

0 

0 

0 

0 

0 

Fault  Seeding 

Mills 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Input  Domain  Based 

Nelson 

3 

wm 

0 

0 

0 

0 

0 

0 

0 

Ho 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Ramamoor thy/Bastani 

3 

0 

0 

■1 

0 

0 

0 

0 

Table  B-32  Subfunction  Ranked  Attributes  6  Metric  Models 


Subfunction  Title:  Provide  Development  Environment 


SUBFUKCTION  ATTRIBUTES,  RANKINGS  6  SCORES 

METRIC  CLASSES  &  MODELS  - 

-  Reli  Surv  Intg  Thru  Usab  Main  Port  Reus  Effc 


HMBMHMLLM 


Time  Between  Failure 

5 

3 

5 

3 

5 

3 

1 

1 

3 

Jelinski/Moranda 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton  (Linear) 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton  (Parabolic) 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Geometric  De-eut) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Hybrid  Geomet  Poiss) 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Littlewood/Ver rail 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Lloyd/Lipow 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Complexity 

Halstead 

5 

0 

0 

0 

0 

3 

0 

0 

0 

McCabe 

5 

0 

0 

0 

0 

3 

0 

0 

0 

Woodward  (Knot  Counts) 

5 

0 

0 

0 

0 

3 

0 

0 

0 

Chen  (Nested  Decision  Stmts) 

5 

0 

0 

0 

0 

3 

0 

0 

0 

Gaffney 

5 

0 

0 

0 

0 

3 

0 

0 

0 

Benyon-Tinker 

Gilb's  (Binary  Decision) 

5 

0 

0 

0 

0 

3 

0 

0 

0 

5 

0 

0 

0 

0 

3 

0 

0 

0 

Chapin's  Q 

5 

0 

0 

0 

0 

3 

0 

0 

0 

Segment-Global  Usage  Pair 
Myer's  (McCabe  extension) 

5 

0 

0 

0 

0 

3 

0 

0 

0 

5 

0 

0 

0 

0 

3 

0 

0 

0 

Hansen's  (McCabe/Halstead) 

5 

0 

0 

0 

0 

3 

0 

0 

0 

Oviedo's  (Data/Ctrl  Flows) 

5 

0 

0 

0 

0 

3 

0 

0 

0 

Failure  Count 

Geol/Okumoto 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Schneidewind 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Geol 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Musa 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Shooman 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Jelinski/Moranda 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Moranda 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto  (Gen  Poisson) 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Brooks/Motley 

5 

3 

0 

0 

0 

0 

0 

0 

0 

IBM  Poisson 

5 

3 

0 

0 

0 

0 

0 

0 

0 

Fault  Seeding 

Mills 

5 

0 

0 

0 

0 

0 

0 

0 

0 

Input  Domain  Based 

Nelson 

5 

0 

0 

0 

0 

0 

mm 

0 

0 

Ho 

5 

0 

0 

0 

0 

0 

0 

0 

Ramamoorthy/Bastani 

5 

0 

0 

0 

0 

0 

0 

Table  B-33  Subfunction  Ranked  Attributes  &  Metric  Models 
Subfunction  Title:  Support  Factory  Test 


SUBFUNCTION  ATTRIBUTES, 

RANKINGS  S, 

SCORES 

Reli 

Surv 

Intg 

Thru 

Usab 

Main 

Port 

Reus 

Effc 

M 

M 

M 

L 

H 

H 

H 

H 

L 

Time  Between  Failure 

3 

3 

3 

1 

5 

5 

5 

5 

1 

Jelinski/Moranda 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton  (Linear) 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton  (Parabolic) 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Geometric  De-eut) 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Hybrid  Geomet  Poiss) 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Littlewood/Verrall 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Lloyd/Lipow 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Complexity 

Halstead 

3 

0 

0 

0 

0 

5 

0 

0 

0 

McCabe 

3 

0 

0 

0 

0 

5 

0 

0 

0 

Woodward  (Knot  Counts) 

3 

0 

0 

0 

0 

5 

0 

0 

0 

Chen  (Nested  Decision  Stmts) 

3 

0 

0 

0 

0 

5 

0 

0 

0 

Gaffney 

3 

0 

0 

0 

0 

5 

0 

0 

0 

Benyon-Tinker 

3 

0 

0 

0 

0 

5 

0 

0 

0 

Gilb's  (Binary  Decision) 

3 

0 

0 

0 

0 

5 

0 

0 

0 

Chapin's  0 

3 

0 

0 

0 

0 

5 

0 

0 

0 

Segment-Global  Usage  Pair 

3 

0 

0 

0 

0 

5 

0 

0 

0 

Myer's  (McCabe  extension) 

3 

0 

0 

0 

0 

5 

0 

0 

0 

Hansen's  (McCabe/Halstead) 

3 

0 

0 

0 

0 

5 

0 

0 

0 

Oviedo's  {Data/Ctrl  Flows) 

3 

0 

0 

0 

0 

5 

0 

0 

0 

Failure  Count 

Geol/Okumoto 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Schneidewind 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Geol 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Musa 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Shooman 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Jelinski/Moranda 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Moranda 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto  (Gen  Poisson) 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Brooks/Motley 

3 

3 

0 

0 

0 

0 

0 

0 

0 

IBM  Poisson 

3 

3 

0 

0 

0 

0 

0 

0 

0 

I 

I 


Fault  Seeding 

Mills 

Input  Domain  Based 


Nelson 

Ho 


Table  B-34  Subfunction  Ranked  Attributes  t  Metric  Models 
Subfunction  Title:  Support  Acceptance  Test 


SUBFUNCTION  ATTRIBUTES, 

RANKINGS  6 

SCORES 

MPTOTP  m &eerc  t  MAnrr  c 

. 

W  M  ■ 

■  "  "  "  " 

Reli 

Surv 

Intg 

Thru 

Usab 

Main 

Port 

Reus 

Effc 

M 

M 

M 

M 

L 

M 

L 

L 

M 

Time  Between  Failure 

3 

3 

3 

3 

1 

3 

1 

1 

3 

Jelinski/Moranda 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton  (Linear) 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton  (Parabolic) 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Geometric  De-eut) 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Hybrid  Geomet  Poiss) 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Littlewood/Ver rail 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Lloyd/Lipow 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Complexity 

Halstead 

3 

0 

0 

0 

0 

3 

0 

0 

0 

McCabe 

3 

0 

0 

0 

0 

3 

0 

0 

0 

Woodward  (Knot  Counts) 

3 

0 

0 

0 

0 

3 

0 

0 

0 

Chen  (Nested  Decision  Stmts) 

3 

0 

0 

0 

0 

3 

0 

0 

0 

Gaffney 

3 

0 

0 

0 

0 

3 

0 

0 

0 

Benyon-Tinker 

Gilb's  (Binary  Decision) 

3 

0 

0 

0 

0 

3 

0 

0 

0 

3 

0 

0 

0 

0 

3 

0 

0 

0 

Chapin's  Q 

3 

0 

0 

0 

0 

3 

0 

0 

0 

Segment-Global  Usage  Pair 
Myer's  (McCabe  extension) 

3 

0 

0 

0 

0 

3 

0 

0 

0 

3 

0 

0 

0 

0 

3 

0 

0 

0 

Hansen's  (McCabe/Halstead) 

3 

0 

0 

0 

0 

3 

0 

0 

0 

Oviedo's  (Data/Ctrl  Flows) 

3 

0 

0 

0 

0 

3 

0 

0 

0 

Failure  Count 

Geol/Okumoto 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Schneidewind 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Geol 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Musa 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Shooman 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Jelinski/Moranda 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Moranda 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto  (Gen  Poisson) 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Brooks/Motley 

3 

3 

0 

0 

0 

0 

0 

0 

0 

IBM  Poisson 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Fault  Seeding 

Mills 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Input  Domain  Based 


Table  Subfunction  Ranked  Attributes  s  Metric  Models 


Subfunction  Title:  Maintain  and  Control  Management  Information 

Database 


SUBFUNCTION  ATTRIBUTES, 

RANKINGS  S 

SCORES 

nr  r  uAnn  c 

Reli 

Surv 

Intg 

Thru 

Usab 

Main 

Port 

Reus 

Effc 

M 

M 

H 

M 

H 

H 

H 

H 

M 

Time  Between  Failure 

3 

3 

5 

3 

5 

5 

5 

5 

3 

Jelinski/Moranda 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton  (Linear) 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton  (Parabolic) 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Geometric  De-eut) 

3 

0 

0 

0 

0 

0 

0 

b 

0 

Moranda  (Hybrid  Geomet  Poiss) 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Li ttlewood/Ver rail 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Lloyd/Lipow 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Complexity 

Halstead 

3 

0 

0 

0 

0 

5 

0 

0 

0 

McCabe 

3 

0 

0 

0 

0 

5 

0 

0 

0 

Woodward  (Knot  Counts) 

3 

0 

0 

0 

0 

5 

0 

0 

0 

Chen  (Nested  Decision  Stmts) 

3 

0 

0 

0 

0 

5 

0 

0 

0 

Gaffney 

3 

0 

0 

0 

0 

5 

0 

0 

0 

Benyon-Tinker 

Gilb's  (Binary  Decision) 

3 

0 

0 

0 

0 

5 

0 

0 

0 

3 

0 

0 

0 

0 

5 

0 

0 

0 

Chapin's  Q 

3 

0 

0 

0 

0 

5 

0 

0 

0 

Segment-Global  Usage  Pair 
Myer's  (McCabe  extension) 

3 

0 

0 

0 

0 

5 

0 

0 

0 

3 

0 

0 

0 

0 

5 

0 

0 

0 

Hansen's  (McCabe/Halstead) 

3 

0 

0 

0 

0 

5 

0 

0 

0 

Oviedo's  (Data/Ctrl  Flows) 

3 

0 

0 

0 

0 

5 

0 

0 

0 

Failure  Count 

Geol/Okumoto 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Schneidewind 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Geol 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Musa 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Shooman 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Jelinski/Moranda 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Moranda 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolverton 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto  (Gen  Poisson) 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Brooks/Motley 

3 

3 

0 

0 

0 

0 

0 

0 

0 

IBM  Poisson 

3 

3 

0 

0 

0 

0 

0 

0 

0 

Fault  Seeding 

Mills 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Input  Domain  Based 

Nelson 

3 

MR 

0 

0 

0 

0 

0 

0 

0 

Ho 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Ramamoo  r  t  hy /Ba  s  t a  n i 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Table  B-36  Subfunction  Ranked  Attributes  i  Metric  Models 


Subfunction  Title:  Management  Information  Tracking 


SUBFUNCTION  ATTRIBUTES, 

RANKINGS  t, 

SCORES 

nXtlKiL.  LLAdoLb  *  nUij£.Mb 

Reli 

Surv 

Intg 

Thru 

Usab 

Main 

Port 

Reus 

Effc 

M 

L 

H 

L 

H 

H 

H 

H 

M 

Time  Between  Failure 

3 

1 

5 

1 

5 

5 

5 

5 

3 

Jelinski/Moranda 

3 

1 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton  (Linear) 

3 

1 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton  (Parabolic) 

3 

1 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Geometric  De-eut) 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda  (Hybrid  Geomet  Poiss) 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Littlewood/Ver rail 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Lloyd/Lipow 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Complexity 

Halstead 

3 

0 

0 

0 

0 

5 

0 

0 

0 

McCabe 

3 

0 

0 

0 

0 

5 

0 

0 

0 

Woodward  (Knot  Counts) 

3 

0 

0 

0 

0 

5 

0 

0 

0 

Chen  (Nested  Decision  Stmts) 

3 

0 

0 

0 

0 

5 

0 

0 

0 

Gaffney 

3 

0 

0 

0 

0 

5 

0 

0 

0 

Benyon-Tinker 

3 

0 

0 

0 

0 

5 

0 

0 

0 

Gilb's  (Binary  Decision) 
Chapin's  0 

3 

0 

0 

0 

0 

5 

0 

0 

0 

3 

0 

0 

0 

0 

5 

0 

0 

0 

Segment-Global  Usage  Pair 
Myer's  (McCabe  extension) 

3 

0 

0 

0 

0 

5 

0 

0 

0 

3 

0 

0 

0 

0 

5 

0 

0 

0 

Hansen's  (McCabe/Halstead) 

3 

0 

0 

0 

0 

5 

0 

0 

0 

Oviedo's  (Data/Ctrl  Flows) 

3 

0 

0 

0 

0 

5 

0 

0 

0 

Failure  Count 

Geol/Okumoto 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Schneidewind 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Geol 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Musa 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Shooman 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Moranda 

3 

1 

0 

0 

0 

0 

0 

0 

0 

Jelinski/Moranda 

3 

1 

0 

0 

0 

0 

0 

0 

0 

Moranda 

3 

1 

0 

0 

0 

0 

0 

0 

0 

Schick/Wolver ton 

3 

1 

0 

0 

0 

0 

0 

0 

0 

Goel/Okumoto  (Gen  Poisson) 

3 

1 

0 

0 

0 

0 

0 

0 

0 

Brooks/Motley 

3 

1 

0 

0 

0 

0 

0 

0 

0 

IBM  Poisson 

3 

1 

0 

0 

0 

0 

0 

0 

0 

Fault  Seeding 

Mills 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Input  Domain  Based 

Nelson 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Ho 

3 

0 

0 

0 

0 

0 

0 

0 

0 

Ramamoorthy/Bastani 

3 

0 

0 

0 

0 

0 

0 

0 

0 

APPENDIX  C 

BROAD  SELECTION  OF  SOFTWARE 
ENVIRONMENT  SUPPORT  TOOLS 


C- 1  through  C-20 


LEGEND 


< COLUMN  HEADING  > 
"Field  Value" 

<APPL-N> 

1.  "SWAT" 

2.  "PSM" 

3.  "OLEA" 

<TYPE> 

<TOOL-NAM> 

<  COMPANY  > 

<CITY> 

<  STATE  > 


DESCRIPTION 


Major  Category 
o  Software  Analysis  Tool 
o  Project  Setup/Monitor 
o  On  Line  Expert  Assistance 

Sub  Category 

(Meaning  Full  Functional 
Descriptors  are  Used) 

Tool  Name 

(Self  Explanator)) 
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APPENDIX  D 


CLASSSIFICATION  TOOLS  MATRIX 


[A  single  copy  of  the  composite  classification  matrix 
used  to  subsequently  categorize  the  tools/environment 
of  Appendix  C  is  separately  bound] 


